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Abstract 


A  novel  approach  to  technical  risk  identification  and  analysis  for  major  weapons  systems  acquisitions  is 
proposed.  It  is  informed  by  the  limitations  of  the  current  risk  matrix.  The  approach  is  to  examine 
representations  of  the  evolving  system  design  to  locate  sources  of  complexity  and  then  inform  the 
program  manager  as  he/she  makes  technical  choices  among  competing  alternatives.  Some  of  the 
alternatives  will  create  more  complexity  and  therefore  more  risk.  The  PM  will  then  be  able  to  balance 
risk  and  reward  at  the  point  of  decision-making,  deciding  to  engage  risk  at  that  moment  by  his/her 
choices.  In  addition,  we  propose  to  rate  or  score  the  contractor  +  government  organizations'  abilities  to 
master  the  complexity  they  have  chosen,  so  that  the  PM  will  know  whether  there  is  a  match  of  product 
complexity  with  organizational  capability. 


Future  work  will  add  dimensions  of  interconnections  and  interdependencies  among  risks,  timing,  delay, 
order  of  risks,  and  uncertainty. 
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Challenges 


"It  is  not  possible  to  know  exactly  how  a  particular  design  will  perform  until  it  is  built.  But  the  product 
cannot  be  built  until  the  design  is  selected.  Thus,  design  is  always  a  matter  of  decision  making  under 
conditions  of  uncertainty  and  risk"  [(Hazelrigg,  1998),  quoted  in  (Deshmukh,  2010),  p.  128]. 


1.  Risk  management  is  many  processes 

Defined  in  the  DoD  Risk  Management  Guide  (2006,  p.  1), 

Risk  is  a  measure  of  future  uncertainties  in  achieving  program  performance  goals  and  objectives 
within  defined  cost,  schedule  and  performance  constraints.  Risk  can  be  associated  with  all 
aspects  of  a  program  (e.g.,  threat,  technology  maturity,  supplier  capability,  design  maturation, 
performance  against  plan,) ....  Risk  addresses  the  potential  variation  in  the  planned  approach 
and  its  expected  outcome. 


Risks  have  three  components: 


•  A  future  root  cause  (yet  to  happen),  which,  if  eliminated  or  corrected,  would  prevent  a 
potential  consequence  from  occurring, 

•  A  probability  (or  likelihood)  assessed  at  the  present  time  of  that  future  root  cause  occurring, 
and 

•  The  consequence  (or  effect)  of  that  future  occurrence. 

A  future  root  cause  is  the  most  basic  reason  for  the  presence  of  a  risk.  Accordingly,  risks  should 
be  tied  to  future  root  causes  and  their  effects. 


Further  risk  management  is  a  number  of  processes: 


Report  No.  SERC-2013-TR-040-2 
Revised  September  3,  201 3 

7 


Contract  Number:  H98230-08-D-01  71 


TO  0030,  RT  040 


UNCLASSIFIED 


Risk 

Identification 


% 


Vl 


Risk 


Risk 

Analysis 


Tracking 


% 


Risk 

Mitigation 

Planning 


Risk 

Mitigation 

Plan  Implementation 


Figure  1.  DoD  Risk  Management  Process  (from  DoD  Risk  Management  Guide,  2006,  p.  4) 


Of  these  processes,  which  are  the  most  important  to  practice,  to  "get  right"?  If  we  want  to  improve  the 
management  of  technical  risks,  on  which  process(es)  should  we  focus? 

Our  conjecture  is  that  Risk  Identification  and  Risk  Analysis  are  key  because,  as  in  the  Figure,  everything 
else  depends  upon  them.  So,  we  are  starting  there. 


2.  Forecasting  risk  is  difficult  and  subjective 

As  listed  above,  program  management  needs  to  collect  and  identify:  future  root  causes,  likelihood,  and 
consequences.  This  is  often,  under  the  best  circumstances,  by  assembling  experts  and  asking  them  to 
converge  on  the  three  components.  It  is  a  group  process  and  based  on  the  extensive  practical 
experience  of  the  expert  panel.  It  is  necessarily  subjective,  based  on  the  memories  of  the  panel 
members  and  their  analogic  reasoning. 

Tversky  and  Kahneman  (see,  for  example,  (Kahneman,  Slovic,  &  Tversky,  1982))  are  celebrated  for  their 
prospect  theory,  which  explains  how  our  biases  interfere  with  our  rational  appraisals.  They  explain  how 
our  subjective  judgments  are  susceptible  to  internal  and  also  normative  forces  that  cloud  our 
perceptions  and  our  reasoning.  Accordingly,  subjective  assessments  of  technical  risk  are  vulnerable  to 
biases. 

And  this  mentions  nothing  about  the  problem  of  using  analogic  reasoning,  which  is  what  expert  team 
members  often  apply.  The  challenge  in  analogical  reasoning  is  that  the  strength  of  the  relevant 
similarities  must  outweigh  the  strength  of  any  significant  dissimilarity.  But  is  that  what  happens  when 
experts  get  together  to  evaluate  risks?  When  one  expert  says,  "This  is  just  like  Project  X,  so  I  can  foresee 
a  risk  of  type  A,"  is  the  analogy  apt?  Is  it  questioned?  [Flere  is  a  list  with  examples  of  errors  by  analogy: 
http://www.skepdic.com/falseanalogy.html] 
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Expert  panels,  in  creating  their  subjective  assessments,  are  susceptible  to  both  biases  and  errors  in 
analogical  reasoning. 


3.  Risk  identification  is  almost  always  post  hoc 

One  part  of  the  Risk  Management  Guide  states,  "Use  a  proactive,  structured  risk  assessment  and 
analysis  activity  to  identify  and  analyze  root  causes, "  (p.  5)  and  others  state,  "Use  the  results  of  prior 
event-based  systems  engineering  technical  reviews  to  analyze  risks  potentially  associated  with  the 
successful  completion  of  an  upcoming  review,"  (p.  5)  and  "During  decomposition,  risks  can  be  identified 
based  on  prior  experience,  brainstorming,  lessons  learned  from  similar  programs,  and  guidance 
contained  in  the  program  office  RMP  [Risk  Management  Plan]."  p.  7. 

Looking  backwards  to  find  risks  is  unassailable,  except  that  it  is  applied  by  judgment  and  analogy  and 
therefore  subject  to  bias  and  error.  For  all  of  the  looking  backwards,  the  purpose  is  to  predict  the  future. 
Where  is  the  justification  of  the  predictions? 


Our  Approach  and  Its  Justification 


Summary 

Our  approach  is  to  characterize  aspects  of  the  technical  products  being  developed  in  a  way  that  would 
inform  a  program  manager  about  to  make  decisions  by  weighing  alternative  technical  courses  of  action. 
We  would  score  or  rank  the  alternatives  based  on  the  relationship  that  each  alternative  would  incur 
future  risk. 


The  result  of  our  research  would  be  a  scorecard,  dash  board,  or  workbench  that  the  program  office 
operates  before  each  major  technical  decision.  The  workbench  would  be  fed  information  about  the 
nature  of  the  product  alternatives  and  based  on  that  would  compute  the  relative  attractiveness  of  each 
option  with  respect  to  incurring  future  risk. 


In  addition  to  examining  characteristics  of  the  product,  the  workbench  would  also  scrutinize  the  match 
between  the  characteristics  of  the  product  and  the  characteristics  of  the  developing  organization,  as  risk 
can  arise  relative  to  an  organization's  capability  to  develop  a  certain  product  alternative. 


As  our  research  progresses,  we  intend  to  add  capabilities  to  take  into  account  the  order  and  timing  of 
decisions,  their  cascade  effects,  and  the  impact/influence  of  uncertainty. 
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We  are  taking  this  approach  for  a  number  of  reasons: 

1.  The  current  method  of  risk  characterization,  measurement,  and  mitigation  has  not  improved 
even  though  the  Department  of  Defense  has  spent  tens  of  millions  of  dollars  on  research  to 
improve  it.  Evidently  the  research  results  have  not  proven  useful  enough  to  change  the 
guidance.  After  all,  much  of  the  DoD  research  investment  has  been  in  Bayesian  methods,  which 
have  been  around  for  almost  200  years  and  still  have  not  found  their  way  into  the  published 
guidance. 

2.  We  have  all  heard  the  remark,  usually  made  informally  by  those  who  see  many  major 
weapons  systems  acquisitions,  that  by  the  time  the  real  issues  become  visible  it  is  very  late  and 
the  effects  have  spread.  We  seek  to  identify  risky  courses  of  action  at  the  time  they  are  being 
considered  for  selection.  This  is  very  early  in  the  unfolding  of  the  systems  development, 
hopefully  in  time  to  take  alternative  steps  if  unaddressed  impacts  are  discovered. 


A.  Objective  Assessment 

Our  approach  does  not  use  any  judgment,  only  objective  measures  of  the  product  and  of  the 
organization's  capability  to  create  the  product.  Accordingly,  we  hope  to  circumvent  the  subjective  biases 
that  can  be  found  in  the  current  DoD  risk  identification  and  analysis  practices. 


B.  Quantifiable  assessment 

We  seek  to  compute  characteristics  about  the  product  alternatives  and  about  organizational  capability, 
so  the  outputs  of  our  analysis  would  be  quantities  that  would  aid  program  management  in  making 
decisions  among  competing  technical  alternatives. 


C.  Aid  in  Decision  Making 

Since  our  approach  is  a  tool  to  be  used  during  decision-making,  we  are  not  taking  a  retrospective  view 
per  se,  but  rather  trying  to  give  the  PM  information  in  order  to  avoid  risk,  that  is,  avoid  encountering  a 
state  of  nature  that  potentially  would  have  unacceptably  high  likelihood  and  consequence. 


D.  Time  is  a  variable,  risks  are  interconnected 

There  is  no  explicit  time  dimension  in  the  current  DoD  risk  management  practice.  We,  on  the  other 
hand,  see  technical  risks  as  largely  interdependent/interconnected,  so  the  order  in  which  the  technical 
decisions  are  considered  matters.  Accordingly,  as  our  research  progresses  we  intend  to  be  able  to 
present  a  program  manager  with  the  options  during  decision-making  of  understanding  the  effects  of 
deferring  or  accelerating  certain  technical  decisions. 


In  addition,  time  plays  another  important  role  in  risk  because  time  delay  between  cause  and  effect 
interferes  with  our  ability  to  connect  the  two,  our  ability  to  reason  about  what  the  root  causes  are  of 
untoward  and/or  unexpected  program  outcomes.  Therefore,  characterizing  the  time-dependent  (that  is. 
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dynamic)  flow  through  the  program  and  technical  product  structures  is  crucial  to  identifying  real,  latent 
causes,  not  just  their  surface  symptoms,  such  as  cost  and  schedule  over-runs. 

E.  Advances  in  Risk  mangement:  Where  to  Look 

A  great  deal  of  work  already  has  been  done  on  improvements  to  risk  identification  and  risk  analysis.  For 
example,  the  DoD  has  sponsored: 

The  Software  Engineering  Institute's  Risk  Program  for  several  decades. 

University  of  Virginia's  Center  for  Risk  Management  of  Engineered  Systems  for  several  decades. 
Research  at  the  Air  Force  Institute  of  Technology  and  Naval  Postgraduate  School  for  decades. 
Research  and  application  at  its  FFRDCs,  such  as  MITRE  and  Aerospace,  for  decades. 


While  the  knowledge  created  at  those  institutions  has  varied,  much  of  it  centered  on  obtaining  a  more 
complete  list  of  risks  and  better  estimates  of  the  likelihood  and  consequence.  Evidently  the  fruits  have 
not  been  powerful  enough  to  change  the  written  DoD  guidance. 


One  could  consult  the  major  defense  contractors,  as  for  decades  they  have  been  actively  managing  the 
risks  of  developing  weapons  systems.  We  approached  a  few  of  them  informally  to  ask  if  they  would 
discuss  with  us  their  risk  management  methods.  They  responded  that  they  considered  their  risk 
management  practices  to  be  competition  sensitive  and  determinative  of  their  commercial  success  and 
would  not  share  them.  We  also  approached  a  few  industrial  firms  and  received  the  same  answer. 


What  about  firms  that  deal  in  risk  every  day,  such  as  insurance  and  investment  businesses?  Flere  the 
final  report  of  a  previous  SERC  research  topic,  valuing  flexibility  (RT-18),  is  dispositive  (Deshmukh,  2010). 


But  what  is  the  connection  between  valuing  flexibility  and  risk?  One  parallel  is  that  both  attempt  to 
characterize  future  uncertainties.  After  all,  flexibility  is  about  responses  to  future  changes,  some 
unplanned.  "Most  approaches  for  valuing  flexibility  depend  on  good  estimation  of  uncertainty. 
Flowever,  estimating  and  characterizing  uncertainty,  even  for  foreseeable  sources  of  [change],  is 
difficult,  especially  in  systems  involving  new  technologies."  p.  24 


Investment  advisors  often  use  the  technique  of  Real  Options  to  find  the  best  investment  among 
alternatives,  akin  to  what  acquisition  program  managers  must  do  at  multiple  points  during 
development.  Flere  are  some  weaknesses  of  Real  Options  in  the  DoD  context  (p.  62): 


"  Financial  options'  assumptions,  such  as  no  arbitrage  condition,  complete  market  condition  and 
infinite  liquidity,  may  not  hold  for  the  non-financial  market. 
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"Without  checking  the  assumption  of  Black-Scholes  model,  using  the  Black-  Scholes  formula  does 
not  make  sense.  For  example,  the  strongest  assumption  of  the  model  is  the  fact  that  uncertainty  can 
be  modeled  in  geometric  Brownian  motion  and  as  a  result  the  distribution  of  future  status  is  [a]  log¬ 
normal  distribution.  If  the  future  environment  cannot  be  modeled  with  this  stochastic  process  and 
distribution,  the  Black-Scholes  model  is  not  valid. 


"Almost  all  real  options  related  literature  assumes  the  risk-neutral  decision  maker  implicitly  or 
explicitly.  This  assumption  need[s]  to  be  check[ed]  in  [the]  risk  management  sense." 


Further,  the  report  continues,  with  some  overlap  with  the  previous  list  (p.  64): 


"  Real  options  must  be  described  in  terms  of  specific  technologies  and  the  systemic  domain  in  which 
they  are  to  be  developed.  Financial  analysis  alone  is  insufficient  to  frame  real  options.  This  is  quite 
difficult,  when  as  yet  undeveloped  technologies  are  under  consideration. 


"Financial  options  are  well-defined  contracts  that  are  tradable  and  individually  valued,  while  real 
options  are  not:  real  options  have  no  contract-specified  exercise  price  of  time.  The  usefulness  of 
valuing  every  potential  program  alternative  that  provides  flexibility  is  not  clear. 


"In  military  procurement  programs,  previous  experiences  associated  with  the  development  of 
similar  technologies  are  not  necessarily  available.  Flence,  valuing  real  options  on  the  basis  of  so 
called  "comparables"  becomes  questionable  because  of  the  absence  of  available  data. 


"Real  options  are  most  often  path-dependent.  Flence,  direct  applicability  of  traditional  financial 
options  methodologies  is  not  appropriate,  as  the  underlying  stochastic  differential  equations  are  not 
necessarily  available. 


"Real  options  in  military  acquisition  programs  are  likely  to  be  highly  interdependent.  Traditional 
financial  option  pricing  methods  fail  here,  again,  because  underlying  stochastic  differential 
equations  may  be  unattainable. 


"In  military  procurement  programs,  there  may  be  no  justifiable  reason  to  accept  the  "no  arbitrage 
assumption".  In  this  case,  general  option  pricing  theory  breaks  down. 


"There  is  typically  no  quantitative  or  qualitative  reason  to  believe  the  real  options  have  uncertainty 
in  price  that  follow  Brownian  motion.  That  is,  unlike  in  financial  markets  where  there  exist  both 
quantitative  and  qualitative  analyses  that  support  by  weak  convergence  in  measure  principles  that 
suggests  a  limiting  Brownian  motion  price  process,  there  is  typically  no  similar  reasoning  supporting 
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the  assumption  of  Brownian  behavior.  Hence,  the  semi-martingale  arguments  leading  to  the 
principal  results  of  general  option  pricing  are  not  applicable." 


And  this  does  not  even  address  what  may  be  the  most  difficult  part  of  the  application  of  Real  Options  in 
the  DoD  context:  the  necessity  to  assess  the  probability  of  each  state  of  nature  in  the  unfolding  of  future 
events.  Investment  analysts  use  historical  information  to  estimate  those  probabilities,  but  there  is  little 
on  which  to  base  estimates  of  weapons  systems  development  probabilities,  especially  of  new 
capabilities. 


In  the  end,  we  cannot  rationally  defend  what  some  other  communities,  above,  use  to  manage  risk 
because  their  assumptions  and  sources  of  data  match  so  little  of  our  situation. 
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Connecting  technical  risk  and  types  of  complexity 


A.  A  FEW  DEFINITIONS 

The  field  of  complexity  is  rich  and  spans  over  the  past  half  century  in  various  fields  of  knowledge  ranging 
from  biological  systems  to  cyber-physical  systems.  As  it  has  been  discussed  by  several  researchers,  an 
strong  correlation  can  be  observed  between  the  complexity  of  the  system  and  various  ranges  of  failures, 
including  catastrophic  failures  (Merry,  1995;  Cook,  2000,  Bar-Yam,  2003). 


The  term  "complexity"  has  several  definitions  and  various  related  aspects  and  characteristics  in  various 
domains  of  knowledge.  We  adopt  the  following  definition: 


Complexity  is  the  potential  of  the  system  to  exhibit  unexpected  behavior  (Willcox,  2011) 


Complex  systems  exhibit  the  potential  for  unexpected  behavior  with  respect  to  variables  of  interest.  The 
potential  can  manifest  itself  in  certain  situations  and  create  the  actual  emergent  behavior  or  stay  hidden 
as  a  potential.  Complex  systems  have  non-linear  interactions,  circular  causality  and  feedback  loops.  They 
may  harbor  logical  paradoxes  and  strange  loops.  Small  changes  in  a  part  of  a  complex  system  may  lead 
to  emergence  and  unpredictable  behavior  in  the  system  (Erdi,  2008).  It  should  be  noted  that  complex 
systems  are  very  different  from  complicated  systems,  and  there  is  a  tendency  for  mistake  in  using  these 
terms  interchangeably.  Complicated  systems  often  have  many  parts,  however  the  interactions  between 
parts  and  subsystems  are  often  well  known  and  linear,  so  they  do  not  show  emergent  or  non-linear 
behavior.  In  contrast,  complex  system  may  or  may  not  have  many  parts,  however,  at  least  one  non¬ 
linear  behavior  of  feedback  loop  exists  in  the  structure  of  the  system  that  drives  emergence  and 
unknown  unknowns  in  the  system. 

The  increased  complexity  is  often  associated  with  increased  fragility  and  vulnerability  of  the  system.  By 
harboring  an  increased  potential  for  unknown  unknowns  and  emergent  behavior,  the  probability  of 
known  interactions  that  lead  to  performance  and  behavior  in  a  complex  system  decreases,  which  in  turn 
leads  to  a  more  fragile  and  vulnerable  system.  That  is,  the  presence  of  complexity  in  a  system,  even  a 
little  complexity,  can  swamp  the  behavior  of  the  familiar,  linear  interactions. 
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Figure  2.  Map  of  the  leading  scholars  and  areas  of  research  in  the  complexity  sciences 
(http://en.wikipedia.org/wiki/Complexity). 


As  can  be  seen  (but  may  not  be  able  to  read!)  from  this  figure,  there  are  many  threads  of  research  and 
many  definitions  of  complexity.  Our  job  is  to  pick  and  choose  among  the  relevant  threads  of  research 
which  can  contribute  to  the  understanding  of  complexity  at  various  milestones  of  acquisition  programs, 
and  identify  the  ones  most  applicable  to  characterizing  technical  risk. 


At  this  early  juncture,  we  can  say  only  that  we  have  not  focused  on  the  following  areas  in  the  diagram 
because  we  think  they  are  not  relevant  to  acquisition  programs: 

Artificial  intelligence  (distributed  neural  networking) 

Agent-based  modeling  (cellular  automata,  genetic  algorithms,  artificial  life,  multi-agent 
modeling 

Case-based  modeling 
New  science  of  networks 
Global  network  society 
Fractal  geometry 

Synergetics/macroscopic  modeling 
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Ecological  systems  theory 
Fuzzy  logic 


We  are  selectively  making  our  way  through  the  remainder  to  assess  suitability  to  characterize  technical 
risk  based  on  what  the  government  sees  during  the  acquisition  life  cycle.  Certain  areas,  such  as 
emergence,  are  potent  metaphors,  but  there  is  a  connotation  among  complexity  researchers  that 
emergence  is  a  property  that  cannot  be  sensed  by  looking  at  components,  so  for  the  moment  we  are 
not  investigating  emergence  further. 

We  are  looking  closely  at  these  kinds  of  complexity,  in  particular: 

•  Structural  -  The  arrangement  of  pieces  and  parts  that  has  loops,  circuits,  so  that  feedback  is 
possible. 

•  Dynamic  -  The  behavior  of  a  system  that  unfolds  as  it  executes.  Here  we  look  for  delays  and 
non-linearities. 

•  Interface,  interconnection  -  How  parts  communicate  and  touch  each  other  and  whether  that 
connection  is  across  a  barrier,  whether  there  is  a  tight  or  loose  connection,  whether  information 
is  hidden  inside  the  components,  and  whether  the  parts  are  of  different  "kinds." 


B.  How  COMPLEXITY  MANIFESTS  AS  TECHNICAL  RISK 

B.1  Mechanisms  and  examples 

One  example  of  risk  is  interconnecting  inhomogeneous  elements.  The  term  is  meant  broadly,  as  it  could 
refer  to  trying  to  connect  two  systems  that  had  never  been  connected  before,  even  though  each  of 
them  was  mature  in  itself.  The  poster-child  for  this  type  of  risk  is  DIVAD,  the  M247  Sergeant  York  "self- 
propelled  anti-aircraft  gun"  (en.wikipedia.org/wiki/DIVAD).  Due  to  the  urgent  need  for  the  capability,  a 
decision  was  made  by  the  Army  to  select  a  design  that  joined  three  commercial  off  the  shelf  systems:  an 
Army  M48  Patton  tank  chassis,  a  radar,  and  a  cannon. 

The  three  particular  commercially  off  the  shelf  systems  selected  by  the  vendor  had  never  been 
connected  and  the  computer  control  system  at  the  heart  had  not  yet  been  developed.  In  the  end,  the 
tank  was  too  slow  to  protect  the  ground  vehicles  it  was  intended  to.  The  radar,  while  off  the  shelf,  was 
off  the  shelf  for  an  airplane!  Airplane  radars  work  internally  by  detecting  movement.  Clearly,  a  tank  in 
the  field  was  not  (always)  in  motion  and  nor  were  its  targets.  The  physical  layout  of  the  radar  with 
respect  to  the  cannon  had  the  cannon  sometimes  getting  into  the  radar's  line  of  sight.  The  tank's  turret 
moved  too  slowly  to  track  realistic  air  targets  because,  after  all,  it  was  never  meant  to.  The  list  went  on. 
And  the  program,  comprised  of  commercial  off  the  shelf  systems,  was  cancelled. 
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How  would  our  analysis  have  identified  these  risks?  By  looking  for  inhomogeneous  interfaces. 


A  second  type  of  complexity  comes  from  feedback  and  delay.  Feedback  itself  is  a  structural 
characteristic:  it  is  a  loop  somewhere  in  the  product  being  developed  or  in  the  organization  creating  the 
product.  And  the  loop  can  amplify  or  dampen  the  signal  passing  through  it,  distorting  the  original  (think 
of  the  child's  game  of  "telephone").  And  the  transit  may  be  delayed  at  points,  which  creates  difficulty  for 
us  humans  to  reason  about  what  causes  the  effects,  the  surface  symptoms,  that  we  see. 


The  field  of  system  dynamics  is  awash  in  examples  of  loops  and  delays,  and  there  is  even  something  of  a 
cottage  industry  in  one  particular  example,  the  Beer  Game\  in  which  a  single  instance  of  a  change  in  a 
single  signal  causes  the  humans  operating  the  game  to  respond  in  a  way  that  causes  oscillation  that 
appears  to  be  unable  to  be  dampened.  All  of  this  due  to  the  (underlying)  structure  of  the  system, 
illustrating  that  structure  produces  behavior. 


The  example  below  comes  from  a  book  on  business  management  (Beer,  1979),  written  to  create  interest 
in  cybernetics.  In  this  example  are  trying  to  construct  a  system  that  has  even  an  output  around  the  value 
0,  given  an  input  single  in  the  form  of  a  regular  sine  wave: 
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Figure  3.  An  output  varying  regularly  about  a  mean  value  that  is  its  target,  showing  the  corrections  at  appear 
necessary  at  each  time  epoch  when  the  measurement  is  made.  (Beer,  1979),  p.  60 


The  approach  is  to  generate  a  -1  when  we  see  a  +1  and  generate  a  +1  when  we  see  a  -1.  Here  is  what 
happens,  according  to  that  rule: 


’  http://www.systemdynamics.orq/products/the-beer-qame/ 
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Figure  4.  Explosive  behavior  induced  by  the  direct  application  of  error  corrections  to  a  system  that  has  reversed  it 
input  states  by  the  time  the  correction  is  applied.  (Beer,  1979),  p.  60 


As  seen,  the  system  is  "exploding."  Why?  Because  the  signal  we  generate  to  correct  for  the  input  is  just 
one  phase  late,  so  instead  of  subtracting  when  it  sees  a  +1,  there  is  a  slight  delay  and  the  negative  signal 
we  generate  supplements  an  already  negative  signal,  making  it  even  more  negative. 


The  important  points  are:  this  is  common  and  is  the  result  of  the  structure  of  the  system,  both  static  and 
dynamic.  Our  methods  of  risk  identification  and  analysis  would  try  to  identify  such  connections  and 
delay. 
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B,2  Risk  and  Complexity  Correlation 

Risk  can  be  defined  as  "a  measure  of  future  uncertainties  in  achieving  program  performance  goals  and 
objectives  within  defined  cost,  schedule  and  performance  constraints. "{US  Department  of  Defense 
(Office  of  the  Undersecretary  of  Defense  (Acquisition,  2006  #1,  p.  1}. 


For  complex  defense  acquisition  programs,  often  various  types  of  risks  exist  that  manifest  themselves  at 
different  times  throughout  the  acquisition  process,  including  system  development.  These  risks  can  be 
technical,  programmatic  or  strategic  in  nature  and  can  result  in  substantial  cost  overruns,  delays, 
performance  issues,  reduced  adaptability  to  changing  requirements  or  even  total  cancellation  of  a 
project.  One  of  the  challenges  with  assessing  risk  using  the  traditional  risk  reporting  matrices  (See 
Figure  5)  for  complex  systems  acquisition  is  that  neither  the  likelihood  nor  the  true  consequence  of  a 
risk  can  be  objectively  established.  For  one,  there  is  substantial  uncertainty  around  the  interactions 
among  different  components  of  a  system  as  well  as  uncertainties  about  how  effectively  various  kinds  of 
risks  can  be  managed  across  a  multiplicity  of  interfaces. 


In  this  research  we  are  proposing  a  fundamentally  different  approach  to  risk  management,  one  that 
looks  at  how  complexities  within  the  technical  and  organizational  realms  result  in  uncertainties  that  can 
ultimately  lead  to  risks  in  the  system.  The  premise  of  this  research  is  that  in  the  realm  of  technical 
project  risk,  it  is  the  complexity  of  the  system  combined  with  the  experience/know-how  of  the 
contractors  that  determines  system  uncertainties  and  the  resulting  risks. 


C  on\t‘«|iifnrr 


Figure  5.  Traditional  Risk  Reporting  Matrix  (US  Department  of  Defense  (Office  of  the  Undersecretary  of  Defense 
(Acquisition,  2006  #1} 
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The  objective  of  this  research  is  to  link  technical  complexity  with  uncertainty  and  risk  across  different 
stages  of  the  acquisition  process,  and  dynamically  quantifying  and  updating  risk  elements  for  decision¬ 
making  on  project  continuation,  modification  or  retirement. 

Complexity  may  be  the  root  cause  of  many  unforeseen  risks.  Program/project  complexity  per  se  can 
generate  negative  consequences  that  may  often  take  the  project  management  team  by  surprise. 
Common  and  advanced  methods  of  risk  modeling,  including,  for  example,  Bayesian  Networks,  cannot 
predict  the  sort  of  emerging  risks  that  manifest  continuous  ripple  effects  that  unfold  one  after  the  other 
almost  for  the  entire  duration  of  the  complex  projects.  This  type  of  intimidating  effect  of  complexity  is 
not  something  one  would  like  to  have  for  the  entire  program  duration,  or  perhaps  at  any  time  during 
the  program,  as  the  effect  is  one  of  being  out  of  control,  or,  indeed,  in  the  control  of  something 
unknown.  Often  the  complexity  manifests  in  risk  and  risk  creates  more  complexity.  This  is  known  as 
complexity-uncertainty  death  spiral.  In  several  case  studies  in  our  previous  research,  we  have  observed 
that  the  increase  in  structural  complexity  increases  the  risk  and  therefore  occurrence  of  the  minor 
undesired  event  (Efatmaneshnik  and  Nilchiani,  2012)(Nilchiani  and  Heydari,  2012).  The  unfolding  of  the 
first  risk  oftentimes  affects  the  structure  of  the  system  in  a  manner  that  increases  the  structural 
complexity.  The  incremental  increase  in  structural  complexity  again  can  contribute  to  the  next  risk  to 
unfold  and  the  spiral  escalation  can  continue.  We  model  this  process  by  hybrid  techniques  and  seek 
techniques  that  tackle  the  root  cause  of  hidden  risks  that  manifest  in  the  form  of  a  set  of  continuously 
mysterious  (no  clear  root  cause)  risks.  There  is  a  very  intricate  relationship  between  structural 
complexity  and  fragility  of  complex  Systems  of  Systems  that  can  be  the  result  of  an  escalation  of  overall 
system  sensitivity,  sometimes  in  a  very  short  time  period  (Figure  6). 


Figure  6.  The  Complexity-Risk  spiral.  Insignificant  uncertainties  and  risks  in  combination  with  structural  complexity 
escalate  into  a  fragile  situation  and  to  a  point  of  no  return  at  which  failure  is  certain. 


Contract  Number:  H98230-08-D-01  71 


Report  No.  SERC-2013-TR-040-2 
Revised  September  3,  201 3 

20 


TO  0030,  RT  040 


UNCLASSIFIED 


System  DSM 


0 


0 


1 


0 


Figure  7.  Structural  complexity  and  risk  (uncertainty)  correlation  in  the  DARPA  F6  program. 

Figure  7  shows  an  example  of  the  structural  complexity  metrics  that  we  defined  and  used  for  the  DARPA 
F6  program  on  fractionated  space  systems  (Nilchiani,  2012).  Fractionated  space  systems  are  a  network 
of  satellites  in  orbit  that  can  consist  of  different  number  of  heterogeneous  satellites  with  various 
architectures  flying  in  formation.  Our  research  has  shown  a  direct  correlation  between  an  increase  in 
structural  complexity  and  how  fast  a  failure  or  risk  in  a  network  of  these  satellites  propagates  (such  as  a 
security  attack  on  one  of  the  satellites  in  the  network).  Figure  8  shows  some  of  the  results  of  the  F6 
simulation  that  connects  the  complexity  measure  of  the  system  to  the  mean  time  to  failure  for  various 
architectures  of  the  fractionated  space  systems. 
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Figure  8.  F6  Simulation  results  showing  that  increased  structural  complexity 
leads  to  shorter  time  to  failure  in  the  system. 


According  to  some  of  our  initial  studies  (Salado  and  Nilchiani,  2012;  Efatmaneshnik  and  Nilchiani,  2012), 
the  results  of  implementing  some  risk  mitigation  plans  can  create  ripple  effects  through  a  project  or 
system,  increase  the  complexity  of  the  system  and  therefore  lead  to  making  the  program  more 
vulnerable  to  known  risks  as  well  as  the  hidden  uncertainties.  Moreover,  the  existence  of  a  minimum  of 
only  three  interrelated  risks  with  significant  correlation  can  lead  to  a  ripple  effect  that  can  remain 
hidden  up  until  the  last  moment,  when  the  negative  consequences  become  fully  developed  and  surface, 
overwhelming  the  system.  Uncovering  these  types  of  hidden  cause  and  effect  relationships  requires 
thorough  structural  monitoring  of  the  system  requirements  and  design  as  early  as  possible  to  uncover  all 
the  dependencies  with  a  very  high  level  analysis. 


Additionally,  as  systems  demonstrate  more  functional  complexity,  they  can  perform  more  sophisticated 
missions.  Flowever,  the  increased  functional  complexity  can  also  produce  an  increased  structural 
complexity  for  systems,  which  in  turn  increases  risks  of  failures.  While  more  complex  functionalities  are 
more  likely  to  deliver  higher  values,  structural  complexity  per  se  is  not  a  positive  attribute.  More 
complex  functions  can  require  that  structurally  complex  structures,  which  one  after  the  other  can  act  in 
unpredicted  ways.  In  essence,  functional  complexity  is  the  driving  force  behind  complexification.  Yet 
structural  complexity  is  a  cost  on  the  system,  because  it  increases  the  possibility  of  dramatic  response  to 
uncertainties,  or  fragility  (Figure  9). 
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Complex  Systems  Engineering  Dilemma 
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Figure  9.  Logical  relationship  between  structural  and  functional  complexity 
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C.  Deriving  Project  Risk  from  Technical  Complexity  and  Contractor  Organizational 
Capabilities 

A  modified  risk  cube  that  looks  at  the  causal  relationships  among  technical  systems  complexity  and 
organizational  capability  in  dealing  with  technical  and  strategic  complexity  is  presented  in  the 
complexity-uncertainty-risk  environment  cube  of  Figure  10. 


Figure  10.  Complexity-Uncertainty-Risk  Environment  (CURE)  Cube 


Here  we  can  explore  the  interrelationships  among  various  aspects  of  technological  complexity  across 
the  acquisition  process  phases  and  explore  the  impact  of  organizational  complexity  (capability)  as  well 
as  strategic  (that  is,  higher  level,  as,  for  example,  at  the  mission  or  campaign  level)  complexity  on 
uncertainties  and  subsequently  risk.  The  unfolding  of  the  technical  complexity  depends  on  the  inherent 
requirements  complexity,  the  system  design,  and  the  capabilities  of  the  contractor  organization(s)  to 
match  their  internal  organizational  complexity  to  manage  the  technical  complexity.  In  our  current 
research  we  are  addressing  only  the  top  row  of  Figure  10,  the  technical  aspects  of  the  system  and  its 
creation  and  testing. 


Figure  11,  below,  illustrates  that  below  the  minimum  required  critical  complexity  a  system  cannot 
perform  the  functions  that  are  expected  and  above  a  maximum  tractable  complexity  level,  the  system 
development  process  can  spiral  out  of  control.  It  is  the  expertise,  know-how  and  experience  of  the 
contractor  organization,  working  with  the  government  acquisition  office  (where  both  use  standard 
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technical  management  processes,  such  as  version  control,  keeping  dependency  graphs  current,  keeping 
design  changes  in  harmony  with  requirements  changes,  etc.),  that  can  keep  the  development  process 
within  the  boundaries  of  these  two  and  stabilize  the  complexity  level  of  a  system.  Thus,  for  the  same 
system  but  different  contractor  +  acquisition  organizations,  the  graph  in  Figure  11  could  have  different 
forms. 

The  key  to  acquisition  risk  management  wiii  therefore  be  to  ensure  a  match  between  the  unfoiding 
technical  complexity  with  the  internal  organization,  know-how  and  expertise  of  the  contractor(s)  + 
acquirer  in  managing  complexity. 


Complexity 


Concept  Program  Technology  Production  and 

Exploration  Definition  Development  Fielding 


Figure  11.  Complexity  evolution  throughout  the  systems  acquisition  lifecycle 


D.  Architectural-level  Complexity  Measures  for  Acquisition  Systems: 

Summary  of  Case  Studies 

The  first  step  therefore  in  transitioning  towards  a  complexity-centric  risk  assessment  is  to  be  able  to 
measure  systems  complexity  over  the  acquisition  process.  As  it  is  possible  that  there  is  no  detailed 
design  in  the  early  stages  of  the  acquisition  process,  the  measurement  of  complexity  has  to  start  at  the 
architectural  (high-level  requirements)  level.  Tables  1  and  2  summarize  the  different  types  and  planes 
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of  acquisition  complexity  at  the  architectural  level.  It  should  be  noted  that  the  following  Tables 
summarize  some  of  the  major  variables  that  contribute  to  the  increased  complexity  of  the  system. 
However  the  list  and  variables  may  not  be  comprehensive  and  in  phase  2  of  the  project,  we  are  aiming 
at  identifying  the  majority  of  variables  that  contribute  to  the  complexity  of  the  system 


Table  1.  Six  types  of  complexity  (Source:  Sheard,  2012) 


Six  Types 

Structural:  Size 

Number  of  elements,  number  of  instances,  total  cost,  total  number  of 
requirements 

Structural:  Connectivity 

Number  of  connections,  density  of  connections,  strengths  of 
relationships,  amount  of  conflict,  distribution  of  connectedness 

Structural:  Inhomogeneity 

Number  of  different  types  of  entities,  number  of  types  of  relationships, 
number  of  different  areas  within  a  space,  diversity  of  sizes  of  elements  or 
contractors  or  stakeholders 

Dynamic:  Short-term 

Existence  of  loops/circuits,  safety-criticality,  tendency  to  blow  up  in 
operational  time  frame,  seriousness  of  consequences  of  a  mishap 

Dynamic:  Long-term 

Evolution  of  purpose  of  an  item,  co-evolution  of  a  variant  and  its 
environment,  how  much  different  the  next  iteration  of  a  system  might  be 

Socio-political 

Fraction  of  stakeholder  interests  that  are  based  on  power,  amount  of 
disagreement  among  stakeholders,  number  of  layers  of  management, 
changes  of  opinion  of  management  or  stakeholders,  number  of  different 
cultures  working  together  on  a  project,  inhomogeneity  of  stakeholder 
utilities. 
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Table  2.  Four  planes  of  acquisition  complexity  (Source:  Sheard,  2012) 


Four  Entities 

[Technical]  System  being  built 

Product,  system,  system-of-systems,  tank,  squadron, 
database,  sensor,  software  algorithm. 

Project  or  organization  doing  the 
building 

Project,  organization,  program,  tasks,  team 

Environment,  both  external  systems  and 
people 

Customers,  buyers,  market,  external  technological  system, 
future  systems  that  need  to  interface  with  product 

Cognitive:  capacity  of  humans  to 
understand,  conceive  of,  build  and 
operate  the  system. 

Learning  curve,  uncertainty,  confusion,  operator  skill  set. 
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E.  Measuring  Architectural  Level  Complexity:  Initial  Explorations 

Based  on  a  comprehensive  literature  and  state  of  the  art  review  we  have  converged  on  the  following 
five  lenses  for  measuring  complexity.  Should  these  prove  to  be  inadequate  for  our  research,  we  will 
devise  new  ones  based  on  our  own  observations  of  systems.  We  will  explore  which  of  the  following 
lenses  of  complexity  measurements  applied  to  an  architecture-level  systems  description  can  dynamically 
predict  acquisition  risk  and  improve  mid-process  decision-support 

1.  Requirement  critical  (algorithmic)  complexity 

2.  Critical  control  path  (cyclomatic)  complexity 

3.  Dynamic  architectural  complexity 

4.  Structural  complexity 

5.  Modified  architectural-structural  complexity 

We  will  then  explore  how  the  experience  of  contractors  +  government  plays  a  role  in  managing  the 
complexity  of  the  system  in  the  acquisition  process. 


1)  Requirement  Critical  Complexity 

Requirement  critical  complexity  refers  to  the  minimum  amount  of  complexity  a  system  needs  to  have  in 
order  to  perform  a  desired  set  of  functions  in  line  with  expressed  requirements.  Based  on  Kolmogorov 
complexity  metric,  it  refers  to  the  minimum  set  of  architectural  level  components  and  linkages  {y}  that 
would  address  requirement  set  {x}.  In  other  words,  the  requirement  critical  complexity  of  a  system  can 
be  expressed  as  the  minimal  systems  architecture  {y}  (minimum  number  and  type  of  components  and 
linkages)  that  would  theoretically  produce  performance  set  {x}. 


The  determination  of  {y}  given  {x}  is  an  important  research  question  and  this  research  will  try  to 
establish  this  threshold  for  various  kinds  of  complex  systems.  The  calculation  of  requirement  critical 
complexity  can  be  done  either  through  modified  structural  complexity  metrics  that  will  be  discussed 
further  in  this  document. 


2)  Critical  Control  Path  Complexity 

Based  on  the  concept  of  cycolmatic  complexity  in  software,  the  critical  control  path  complexity  metric 
measures  the  number  of  linearly  independent  control  paths  through  a  systems  architecture  graph.  This 
number  changes  as  the  architecture  (or  the  resulting  design)  changes  over  time  and  is  estimated  by  the 
following  equation: 


CCP{t)  =  ^  n{t,  i)  —  Z(t,  i)  +  p{t,  i) 
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where  CCP{t)  is  the  critical  control  path  complexity  at  time  t,  n(t,i)  is  the  number  of  nodes  in  the 
connected  graph  of  the  architecture  expression  for  module  (i),  /  is  the  number  of  linkages  at  time  t  in 
module  (i)  and  p  is  the  number  of  distinct  connected  components  in  the  architectural  flow  graph.  And 
the  sum  is  over  all  modules. 


3)  UML-based  Five-Views  Dynamic  Architectural  Complexity  (Lankford) 

Rather  than  a  single  number,  the  UML-based  Five  Views  Dynamic  Architectural  complexity  metric  allows 
the  measurement  of  various  system  complexity  metrics  over  time. 

The  five  views  complexity  vector  is  calculated  as  follows: 


C5l^(t)  = 


^class(,^>  0 
^process  (t,  0 
^component  (pi  0 
^interfacesO'i  0 
^patternsO'i  0 


Where: 


Within 

Cciassi^,  i)=Number  of  classes  and  objects  within  each  module  at  time  t 
Cprocessi^'  0  =Number  of  processes  and  threads  within  each  module  at  time  t 
(^component 0  =Number  of  components  =  the  number  of  nodes 
C inter faceiP,  0  =Number  of  interfaces  between  each  of  these 
Cpatternu^f'  0  =Number  of  identifiable  design  patterns  within  each  module 


4)  Simple  Structural  Complexity  (Meyer) 

Simple  structural  complexity  can  provide  an  easy  to  calculate  way  to  capture  how  the  complexity  of  a 
system  is  changing,  by  calculating  changes  in  the  number  of  parts  (or  sub-systems  or  systems),  types  of 
parts  and  number  of  interfaces  over  time. 


^structurali.^f  0  “  ^  0  ^  0 

Where 

Np(t,  i)  =Number  of  parts/subsystems  in  subsystem/system  ;  at  time  t 
Ny(t,  0=  Types  of  parts/subsystems  in  subsystem/system  /  at  time  t 
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Nx{t,  i)=  Number  of  interfaces  in  subsystem/system  ;  at  time  t 


5)  Modified  Architectural-Structural  Complexity  (MASC) 

The  modified  architectural-structural  complexity  is  the  most  comprehensive  measure  of  architectural 
complexity,  taking  into  account  size,  type,  interconnections  and  interfacial  complexity  of  architectural 
modules  into  consideration.  It  is  based  on  Kinnunen  (2000).  Modifying  the  simple  architectural 
complexity  equation  for  MASC  we  get: 


CmasM  =  ^V[(iVp(t)  X  Ny{t,  0]  X  X  Nabjit)  X  Napit)]  X 

Where  the  arguments  are  respectively: 

•  Number  of  distinct  types  of  objects/components 

•  Number  of  objects  within  each  type 

•  Number  of  processes/functionalities  affecting  an  object 

•  Number  of  objects/components  affecting  a  process 

•  Number  of  operations  per  process 

•  Number  of  interfaces  weighted  with  the  interface  complexity  multiplier  (ICM)  (related 
to  the  integration  readiness  levels  (IRLs)  between  different  systems/subsystems). 

It  should  be  noted  that  these  five  types  of  architectural  level  complexity  measures  are  our  initial 
exploration  of  the  relevant  complexity  measures  of  the  technical  system.  Our  research  team  may  have 
to  define  novel  measures  based  on  the  existing  literature  on  complexity  that  may  be  more  useful  for 
different  milestones  of  an  acquisition  program,  and  in  particular  characterizing  the  dynamic  behavior  of 
the  architecture. 


C.  The  Fit  Between  Technical  Risk  of  the  Product  and  an  Organization's  Capability  to  Manage  it 


What  accounts  for  one  enterprise  being  able  to  create  a  complex  product  and  another  not?  The  primary 
conjecture,  attributed  to  Ashby  (Ashby,  1961),  is  that  the  successful  enterprise  that  can  construct  a 
complex  product  has  enough  "variety"  (he  called  it  requisite  variety)  in  the  way  it  is  organized  and 
applies  its  resources.  Variety  is  diversity,  ability  to  react  to  various  problems  and  opportunities,  including 
unexpected  ones. 


Perhaps  one  of  the  most  vivid  illustrations  of  variety  in  this  context  was  during  the  Apollo  13  manned 
space  flight  in  1970,  when  an  oxygen  tank  aboard  exploded,  limiting  power,  causing  loss  of  cabin  heat, 
reducing  the  availability  of  potable  water,  and  increasing  the  concentration  of  carbon  dioxide  in  the 
cabin  air.  It  was  the  mounting  concentration  of  carbon  dioxide  that  proved  most  troubling,  as  the 
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astronauts  would  die  of  lack  of  oxygen  if  it  were  not  reversed.  A  team  on  the  ground  was  assembled  and 
given  the  task  of  figuring  out  how  to  create  a  carbon  dioxide  removal  system,  given  the  constraints  on¬ 
board.  That  the  ground  team  succeeded  was  a  tribute  to  its  variety,  its  diversity  of  thought,  as  it  quickly 
suggested  and  tested  numerous  options. 


One  of  the  biggest  challenges  in  using  variety  to  characterize  organizations  is  that  it  is  so  difficult  to 
observe,  to  measure.  Two  authors  (Beer,  1979;  Jaques,  2006)  have  suggested  antidotes  to  this  and  we 
are  exploring  their  methods. 


D.  The  Places  of  Case  Studies  and  Quantitative  Data 

We  are  seeking  to  know  what  programs  and  organizations  are  "made  of"  that  might  inform  the 
identification  of  risk.  Our  premise  is  that  complexity  is  a  major  indicator  of  risk.  In  order  to  validate  or 
invalidate  the  premise  we  need  data.  The  most  convincing  data  would  be  numeric,  quantitative  that 
showed  the  relationship  between  product  complexity,  say,  and  risk.  If  that  data  are  not  available,  then 
we  might  use  case  studies. 


Since  we  do  not  yet  have  quantitative  data,  we  have  indeed  been  reading  case  studies,  supplied  to  us  by 
the  deep  reservoir  provided  by  our  colleague.  Dr.  Gary  Witus,  at  Wayne  State  University.  At  one  stage 
he  supplied  15  cases,  some  with  multiple  artifacts.  Dr.  Mostashari  read  them  and  in  the  end  was  not 
able  to  deduce  anything  general. 


Dr.  Witus  responded  to  our  request  for  additional  case  studies  and  we  have  not  yet  had  a  chance  to 
absorb  them,  and  it  is  a  priority  for  our  next  steps. 


At  some  point  -  earlier  is  better  -  we  are  going  to  need  access  to  quantitative  data  that  will  help  us 
confirm  or  deny  the  connection  between  some  measure  of  complexity  and  technical  program  risk.  This, 
too,  is  a  priority  for  the  next  steps. 


Both  case  studies  and  access  to  quantitative  program  information  will  help  us  steer  where  to  look 
deeper,  help  us  consider  what  programs  are  made  of.  In  the  end,  it  is  possible  that  programs  do  not 
collect  the  measures  of  complexity  that  we  think  are  the  most  indicative  of  risk,  so  we  will  have  to  work 
with  programs  on  a  pilot  basis  to  install  new  measures  and  assess  the  ability  of  those  measures  to 
predict  technical  risk. 
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Examples  and  Some  Case  Studies 


Concept  Demonstration:  Complexity  and  Risk 

One  of  the  easy  ways  to  characterize  complexity  for  a  system  at  an  architectural  level  is  to  analyze  the 
number  of  different  interactions  between  the  different  subsystems. 

hueracnons„^  = 

nvn  ^ 


Interactions  can  be  spatial,  material,  energy  and/or  information.  If  we  look  at  aircraft  designs  over  the 
years  we  can  see  how  avionics  has  become  more  complex. 


Figure  12.  Increasing  avionics  complexity  (dimensionless)  over  the  years  (Source:  Diterick,  2010) 

Analyzing  the  relationship  between  cost  and  development  schedule  in  154  DoD  projects,  McNutt  (2000) 
estimates  the  relationship  between  cost  and  complexity  to  be  estimated  by  the  following  equation: 

DevelopnieniCosf  =  (0.03  x  DeydopmetvThuc^^,^^  + 1 .36)^ 

Where  the  development  cost  is  in  millions  of  USD. 
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Case  Studies 

An  analysis  of  the  following  31  acquisition  programs  with  a  calibrated  complexity  metric  shows  the 
following  correlation: 


Figure  13.  Cost  overrun  as  a  function  of  log  calibrated  architectural  systems  complexity 
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Figure  14.  Cost  overrun  and  schedule  slips  for  different  types  of  weapons  systems.  Most  cost  overruns  occur  for 
ship  systems,  while  most  schedule  slips  happen  for  aircraft.  Avionic  systems  have  had  a  good  track  record  of 
beating  both  cost  and  schedule  plans. 
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Program  Nai 


Total  Program  ( 


Type  of  Systet^ 


C-130 

$6,204 

Aircraft 

Boeing 

252 

0 

High  TRL 

E2-D  Advanced 
Hawkeye 

$17,747 

Aircraft 

Northrop  Gruman 

20.3 

43.2 

Medium  TRL 

F-35 

$326,535 

Aircraft 

Lockheed  Martin 

78.2 

N/A 

Low  TRL 

FAB-T 

$4,688 

Aircraft 

Boeing 

29.1 

35 

Medium  TRL 

Global  Hawk 

$12,812 

Aircraft 

Northrop  Gruman 

172.2 

127.3 

Low  TRL 

Grey  Eagle 

$5,159 

Aircraft 

General  Atomics 

-18 

N/A 

High  TRL 

HC-130 

$13,091 

Aircraft 

Lockheed  Martin 

-5.1 

N/A 

High  TRL 

MQ-4C  UAV 

$13,052 

Aircraft 

Northrop  Gruman 

1.6 

0 

High  TRL 

P-8A  Poseidon 

$32,969 

Aircraft 

Boeing 

0.1 

0 

High  TRL 

Reaper  UAV 

$11,919 

Aircraft 

General  Atomics 

18.9 

19 

Medium  TRL 

Excalibur  Guided 
Artillery 

$1,781 

Artillery 

Raytheon 

282.4 

27.2 

Medium  TRL 

IDECOM 

$821 

Avionic  System 

ITT  Electronics 

-0.5 

-8.5 

High  TRL 

Joint  Precision- 

Apparoach  and 
Landing  System 

$26,575 

Avionic  System 

Raytheon 

-2.9 

2.7 

High  TRL 

Airbrone  and 

Tactical  Radio 

System 

$8,160 

Communication  System 

Lockheed  Martin 

0.1 

13.8 

Medium  TRL 

Joint  Tactical 
Radio  System 
Handheld 

$8,358 

Communication  System 

General  Dynamics 

1 

22.4 

Medium  TRL 

Mobile  User 
Objective  System 

$6,978 

Communication  System 

Lockheed  Martin 

3.8 

28.9 

Medium  TRL 

Navy  Multi-band 
Terminal 

$1,214 

Communication  System 

Raytheon 

-11.2 

0 

High  TRL 

Warfighter 

information 

Network  Tactical 

$6,052 

Communication  System 

General  Dynamics 

8.6 

42 

Medium  TRL 

Apache  block  IIIA 

$10,737 

Helicopter 

Boeing 

39.7 

3.8 

High  TRL 

CH-53 

$22,439 

Helicopter 

Sikorsky 

5.7 

32 

High  TRL 

AGM88E 

$1,902 

Missile 

ATK  Missile  Systems 

10.9 

22.4 

High  TRL 

Army  Integrated 
Air  and  Missile 

Defense 

$5,529 

Missile 

Northrop  Gruman 

9.9 

1.3 

High  TRL 

Joint  Land  Attack 

Cruise  Missile 

Defense 

$7,858 

Missile 

Raytheon 

18 

6.2 

Medium  TRL 

Standard  Missile 

RAM 

$6,297 

Missile 

Raytheon 

10.5 

25.3 

Medium  TRL 

CVN  78 

$33,994 

Ship 

Huntington  Ingalls 

-4.4 

13.1 

High  TRL 

DDG  1000 

$20,985 

Ship 

BAE  Systems 

543 

73 

Low  TRL 

Joint  Highspeed 
Vessel 

$3,674 

Ship 

Austral  USA 

1 

4.2 

High  TRL 

LHA  Replacement 
Assault  Ship 

$10,096 

Ship 

Huntington  Ingalls 

5.8 

13 

High  TRL 

LCS 

$32,867 

Ship 

Lockheed  Martin 

76 

183 

Low  TRL 

GPS  Hi 

$4,210 

Space  System 

Lockheed  Martin 

6.8 

N/A 

Medium  TRL 

Space-Based  IR 
System  (SBIRS) 

$18,266 

Space  System 

Lockheed  Martin 

231.2 

N/A 

Low  TRL 

Table  3.  A  selection  of  case  studies  of  DoD  acquisition  program  and  their  percentage  of  cost  and  schedule 

overruns. 
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Average  Program  Size 
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■  Average  Program  Sire 


Figure  15. Average  program  size  (total  program  cost)  of  case  studies  explored  in  this  research  (in  million  $) 


Additional  Case  Studies  (Not  included  in  Complexity  Analysis) 

The  following  are  additional  case  studies  the  team  looked  at,  but  most  were  suffering  from  program 
complexity  rather  than  technical  complexity. 


A-10  Thunderbolt  II  (Aircraft) 

Acquisition  Organization:  U.S.  Air  Force 

Risks  and  Weaknesses 

■  Technical:  Concurrent  development  of  a  new  technology  (the  GAU-8/A  gun  system)  and  the 
aircraft  at  the  same  time  with  the  aircraft  architecture  revolving  around  the  armament  system 
created  delays  in  the  acquisition  process.  Also  the  original  structural  design  proved  inadequate 
for  the  design  life,  and  even  fixes  during  production  were  inadequate  for  all  but  the  latest 
aircraft  produced. 

■  Programmatic:  Overlooked  problems  associated  with  production  readiness  and  contractor 
financial  stability  did  not  go  away  and  had  to  be  resolved  far  too  late  in  the  development 
program.  Additional  problems  included  loss  of  the  Original  Equipment  Manufacturer  (OEM),  on- 
again/off-again  decisions  to  retire  the  A-10,  unstable  funding  for  inspection  and  repair,  and 
major  personnel  disruptions  resulting  from  a  BRAC  decision.  Critical  "health  of  the  fleet" 
structural  inspections  were  not  performed  during  sustainment,  and  a  subsequent  repair 
program  failed  to  provide  the  desired  level  of  life  extension. 

Strengths 

Close  attention  to  key  mission  characteristics  (lethality,  survivability,  responsiveness,  and 
simplicity)  allowed  the  concept  formulation  and  subsequent  system  design  to  result  in  an 
effective  CAS  aircraft,  and  design-to-cost  goals  kept  the  government  and  contractor  focused  on 
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meeting  the  critical  requirements  at  an  affordable  cost.  The  A-10  did  not  meet  all  its  cost  goals, 
but  it  came  much  closer  to  them  than  most  major  defense  development  programs  did  in  that 
time  frame  or  since  then. 

Complexity  Factors  Leading  to  Risk 

•  Low  TRL  technology  at  core  of  systems  architecture  (Interface  complexity) 

•  Requirement  changes  rendering  architecture  inadequate  (Requirement  Complexity) 

Contractor  technical  and  financial  capability  (Organizational  Requisite  Complexity) 

Source:  A-10  Thunderbolt  II  (Warthog)  SYSTEMS  ENGINEERING  CASE  STUDY ,  Air  Force  Center  for 

Systems  Engineering 


C-5A  Galaxy  (Aircraft) 

Acquisition  Organization:  U.S.  Air  Force 

Risks  and  Weaknesses 

■  Technical:  A  Weight  Empty  Guarantee  was  included  in  the  specification  as  a  performance 
requirement  and  in  the  contract  as  a  cost  penalty  for  overweight  conditions  of  delivered  aircraft. 
The  aircraft  Weight  Empty  Guarantee  dominated  the  traditional  aircraft  performance 
requirements  (range,  payload,  etc.),  increased  costs,  and  resulted  in  a  major  shortfall  in  the  wing 
and  pylon  fatigue  life.  The  stipulation  of  a  Weight  Empty  Guarantee  as  a  performance 
requirement  had  far-reaching  and  significantly  deleterious  unintended  consequences. 

■  Programmatic:  The  Total  Package  Procurement  Concept  (TPPC)  employed  by  the  government 
required  a  fixed-price,  incentive  fee  contract  for  the  design,  development,  and  production  of  58 
aircraft.  It  included  a  clause  giving  Total  Systems  Performance  Responsibility  (TSPR)  to  the  prime 
contractor.  TPPC  was  invented  to  control  costs,  but  it  was  the  underlying  cause  of  the  cost 
overrun  and  limited  the  number  of  aircraft  purchased  under  the  original  contract 

Strengths 

The  process  for  developing  and  documenting  the  system  performance  requirements  involved 
the  User  (warfighter),  planners,  developers,  and  technologists  from  both  the  government  and 
industry  in  a  coordinated  set  of  trade  studies.  It  resulted  in  a  well-balanced,  well-understood  set 
of  requirements  that  fundamentally  remained  unchanged  throughout  the  program. 

Complexity  Factors  Leading  to  Risk 

•  Detrimental  hard  requirement  with  cascading  effect  on  mission  critical  requirements  and 
architectural  design  (Requirement  complexity) 

•  Weight,  wing  and  pylon  design  conflict  (Interfacial  complexity) 

•  Faulty  procurement  concept  (Organizational  Process  Complexity) 

Source:  C-5A  Galaxy  Systems  Engineering  Case  Study,  Air  Force  Center  for  Systems  Engineering 
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F-lll  (Aircraft) 

Acquisition  Organization:  U.S.  Air  Force  and  U.S.  Navy 

Risks  and  Weaknesses 

■  Technical:  The  F-lll  acquisition  process  suffered  from  a  nearly  impossible  multi-role/multi- 
service  requirement  specification,  and  a  protracted  development  cycle  in  which  numerous 
serious  technical  problems  had  to  be  identified  and  corrected.  Of  the  1,726  total  aircraft  buy 
that  had  originally  been  planned  in  1962,  only  562  production  models  of  seven  different  variants 
were  completed  when  production  ended  in  1976.  The  F-lll,  like  any  complex  weapon  system 
development  program,  which  provides  new  war-fighting  capability,  had  areas  of  risk  or 
deficiency  that  came  to  light  during  RDT&E  even  though  there  was  perceived  low  risk  in  the 
design.  The  F-lll  development  program  introduced  concurrency  (overlap)  between  design 
validation/verification  and  production  to  accelerate  program 

■  Programmatic:  Systems  Architecture  and  Design  Trade-Offs  were  not  performed  to  achieve  an  F- 
111  design  that  was  balanced  for  performance,  cost  and  mission  effectiveness  (including 
survivability)  and  the  attendant  risk  and  schedule  impacts.  The  F-lll  suffered  from  poor 
communications  between  the  Air  Force  and  Navy  technical  staffs,  and  from  over-management 
by  the  Secretary  of  Defense  and  the  Director,  Defense  Research  and  Engineering,  and  it  came 
under  intense  congressional  scrutiny,  which  restricted  the  System  Program  Office  (SPO)  Director 
from  applying  sound  systems  engineering  principles. 

Complexity  Factors  Leading  to  Risk 

•  Impossible  requirements  with  severe  conflicts  (Requirement  complexity) 

•  Inadequate  verification  and  validation  (Organizational  Process  Complexity) 

•  Multi-agency  acquisition  process  (Organizational  Process  Complexity) 

•  Sociopolitical  sensitivity  (Organizational  Process  Complexity) 

Source:  Fill  Systems  Engineering  Case  Study,  Air  Force  Center  for  Systems  Engineering 
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AGM-88E  Advanced  Anti-Radiation  Guided  Missile  (AARGM) 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  ATK  Missile  Systems 

As  of 

Latest 

Percent 

Company 

07/2003 

06/2011 

change 

Program  office:  Patuxent  River.  MD 

Research  arKi  development  cost 

$637.2 

$722.2 

13.4 

Funding  needed  to  complete: 

Procurement  cost 

$963.6 

$1,180.0 

22.5 

R&D:  SO.O  million 

Total  program  cost 

$1,600.7 

$1,902.3 

18.8 

Procurement:  $1,319.6  million 

Program  unit  cost 

$.894 

$991 

10.9 

Total  fundirtg:  $1,319.6  miOion 

Total  quantities 

1,790 

1,919 

7.2 

Procurement  quantity:  1 ,767 

Acquisition  cycle  time  (months) 

85 

104 

22.4 

Apache  Block  IIIA 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Pnme  contractor  Boeing 

As  of 

Latest 

Percent 

Program  office:  Huntsville,  AL 

08/2006 

12/2010 

change 

Funding  needed  to  complete: 

Research  and  devetopmetit  cost 

$1,155.6 

$1,640.3 

41.9 

R4D:  $706.8  million 

F*rocurement  cost 

$6,086.9 

$9,096.8 

49.4 

Procurement:  $8,363.7  million 

Total  program  cost 

$7,242.5 

$10,737.0 

48.3 

Total  funcRng:  $9,070.6  minion 

Program  unit  cost 

$12,031 

$16,803 

39.7 

Procurement  quantity:  610 

Total  quantities 

602 

639 

6.1 

Acquisition  cycle  time  (months) 

79 

82 

3.8 

Ttie  lalest  cost  ana  quaraaes  Do  not  nouae  tn«  57  ne«-ouiKl  neiicopters  g'jt  are  Being  acqured 

unoer  me  A83e  major  ae'enee  acquisition  orogtam. 

Army  Integrated  Air  and  Missile  Defense 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Pnme  contractor  Northrop  Gmmman 

As  of 

Latest 

Percent 

Space  &  Mission  Systems  Corp. 

12/2009 

08/2011 

change 

Program  office:  Huntsville,  AL 

Research  and  development  cost 

$1,595.2 

$2,019.8 

26.6 

Funding  needed  to  complete: 

Procurement  cost 

$3,433.4 

$3,509.0 

2.2 

R&D:  $1,370.5  million 

Total  program  cost 

$5,028.6 

$5,528.8 

9.9 

Procurement:  $3,509.0  million 

Program  unit  cost 

$16,988 

$18,678 

9.9 

Total  funding:  $4,879.5  million 

Total  quantities 

296 

296 

0.0 

Procurement  quantity:  285 

Acquisition  cycle  tme  (months) 

80 

81 

1.3 

Contract  Number:  H98230-08-D-01  71 


Report  No.  SERC-2013-TR-040-2 
Revised  September  3,  201 3 

39 


TO  0030,  RT  040 


UNCLASSIFIED 


C-130  Avionics  Modernization  Program 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Boeing 

As  of 

Latest 

Percent 

Program  office;  Wright-Patterson  AFB, 

07/2001 

12/2010 

change 

OH 

Research  and  development  cost 

$775.4 

$1,948.3 

151.3 

Funding  needed  to  complete; 

Procurement  cost 

$3,356.8 

$4,256.0 

26.8 

R4D;  S42.2  r«llion 

Total  program  cost 

$4,132.3 

$6,204.3 

50.1 

Procurement;  S3, 890.4  million 

Program  unit  cost 

$7,962 

$28,074 

252.6 

Total  funding;  S3.932.6  million 

Total  quantities 

519 

221 

-57.4 

Procurement  quantity;  208 

Acquisition  cycle  time  (months) 

NA 

NA 

NA 

CH  53-K  Heavy  Lift  Replacement 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

F>rime  contractor  Sikorsky  Aircraft 

As  of 

Latest 

Percent 

Corporation 

12/2005 

08/2011 

change 

F>rogram  office;  Patuxent  River.  MD 

Research  and  development  cost 

$4,378.9 

$6,058.1 

38.3 

Fundirig  needed  to  complete: 

Flocurement  cost 

$12,178.3 

$16,381.7 

34.5 

R&D:  $3,252.9  million 

Total  program  cost 

$16,557.1 

$22,439.9 

35.5 

Procurement:  $16,381.7  million 

Program  unit  cost 

$106,136 

$112,199 

5.7 

Total  funding:  $19,634.6  million 

Total  quantities 

156 

200 

28.2 

Procurement  quantity;  196 

Acquisition  cycle  time  (months) 

119 

157 

31.9 

CVN  78  Class 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Huntington  Ingalls 

As  of 

Latest 

Percent 

Industries-Newport  News 

04/2004 

08.'2011 

change 

Program  office:  Washington,  DC 

Research  and  development  cost 

$4,803.3 

$4,646.8 

-3.3 

Funding  needed  to  complete: 

Procurement  cost 

$30,770.8 

$29,346.8 

-4.6 

R&D:  $827.7  nvllion 

Total  program  cost 

$35,574.1 

$33,993.6 

-4.4 

Procurement:  $16,540.9  million 

Program  unit  cost 

$11,858,040 

$11,331,185 

■4.4 

Total  funding;  $17,368.6  million 

Total  quantities 

3 

3 

0.0 

Procurement  quantity:  2 

Acquisition  cycle  time  (months) 

137 

155 

13.1 
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DDG  1000  Destroyer 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  BAE  Systems,  Bath 

As  of 

Latest 

Percent 

Iron  Worlts,  Huntington  Ingalls 

01/1998 

08/2011 

change 

Industries,  Raytheon 

Research  arxt  development  cost 

$2,277.9 

$10,378.4 

355.6 

Program  office:  Washington.  DC 

Procurement  cost 

$32,522.1 

$10,607.2 

-67.4 

Funding  needed  to  complete: 

Total  program  cost 

$34,800.0 

$20,985.6 

-39.7 

R&D:  $995.6  million 

Program  unit  cost 

$1,087,500 

$6,995,214 

543.2 

Procurement:  $1,418.1  million 

Total  quantities 

32 

3 

-90.6 

Total  funding:  $2,413.6  million 

Acquisition  cycle  time  (months) 

128 

222 

73.4 

Procurement  quantity:  0 

E2-D  Advanced  Hawkeye 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Northrop  Grumman 

As  of 

Latest 

Percent 

FYogram  office:  Patuxent  River.  MD 

06/2003 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$3,841.0 

$4,537.9 

18.1 

R&D:  $367.8  nxllion 

Procurement  cost 

$10,911.1 

$13,167.1 

20.7 

Procurement:  $10,874.7  million 

Total  program  cost 

$14,752.0 

$17,747.3 

20.3 

Total  funding:  $11,257.5  million 

Program  unit  cost 

$196,694 

$236,630 

20.3 

Procurement  quantity:  60 

Total  quantities 

75 

75 

0.0 

Acquisition  cycle  time  (months) 

95 

136 

43.2 

Excalibur  Precision  Guided  Artillery 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Raytheon 

Program  office:  Rcatinny  Arsenal,  NJ 
Funding  needed  to  complete: 

Research  and  development  cost 

As  of 
02/2003 
$765.5 

Latest 

08^2011 

$1,068.3 

Percent 

change 

39.6 

R&D:  $50.2  million 

Procurement  cost 

$4,010.8 

$712.1 

-82.2 

Procurement:  $234.9  million 

Total  program  cost 

$4,776.2 

$1,780.5 

-62.7 

Total  fundirtg:  $285.1  million 

Program  unit  cost 

$.062 

$.238 

282.4 

Procurement  quantity:  3,455 

Total  quantities 

76,677 

7.474 

-90.3 

Acquisition  cycle  time  (months) 

Ibtai  quanmes  induoe  3.XS5  ncrenen:  le  pnjecqies 

136 

173 

27.2 
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F-35  Lightning  II 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Lockheed  Martin, 

As  of 

Latest 

Percent 

Pratt  and  Whitney 

10/2001 

12/2010 

change 

Program  office:  Arlington,  VA 

Research  and  development  cost 

$38,976.7 

$58,387.6 

49.8 

Funding  needed  to  complete: 

Procurement  cost 

$172,921.4 

$267,595.6 

54.7 

RSJD:  $10,117.8  million 

Total  program  cost 

$213,708.2 

$326,535.2 

52.8 

Procurement:  $245,676.5  million 

Program  unit  cost 

$74,567 

$132,900 

78.2 

Total  funding:  $255,970.4  million 

Total  quantities 

2,866 

2.457 

-14.3 

Procurement  quantity:  2,353 

Acquisiton  cycle  time  (months) 

116 

TBD 

NA 

.atest  column  Does  not  ruily  renecttnoresouctureo  jSF  program.  CocIs  are  expecteo  to  grow  and  me 

schedue  vnii  De  extended 

Family  of  Advanced  Beyond  Line  of  Sight  Terminals  (FAB-T) 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Boeing 

As  of 

Latest 

Percent 

Program  office:  Hanscom  AFB,  MA 

12/2006 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$1,537.1 

$2,338.7 

52.2 

R&D:  $571.4  million 

Procurement  cost 

$1,651.4 

$2,349.6 

42.3 

Procurement:  $2,338.6  million 

Total  program  cost 

$3,188.5 

$4,688.3 

47.0 

Total  funding:  $2,910.0  million 

Program  unit  cost 

$14,762 

$19,058 

29.1 

Procurement  quantity;  216 

Total  quantities 

216 

246 

13.9 

Acquisition  cycle  time  (months) 

129 

174 

34.9 

Tn«  .a'.ec  cost  data  do  n«  reflect  m«  current  cos:  or  me  program.  A  new  acgusipon  program  oaseilne 

nas  not  yet  Deen  approved. 

Global  Hawk 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Northrop  Grumman 

As  of 

Latest 

Percent 

Program  office;  Wright-Patterson  AFB, 

03/2001 

10/2011 

change 

OFI 

Research  arxl  development  cost 

$1,041.6 

$4,769.3 

357.9 

Funding  needed  to  complete: 

Procurement  cx3®t 

$4,318.8 

$7,877.4 

82.4 

R&D:  $1,657.1  rrtllion 

Total  program  cost 

$5,392.0 

$12,811,6 

137.6 

Procurement:  $3,098.9  million 

Program  unit  cost 

$85,588 

$232,938 

172.2 

Total  funding;  $4,789.4  million 

Total  quantities 

63 

55 

-12.7 

Procurement  quantity:  13 

Acquisition  cycle  time  (months) 

55 

125 

127.3 
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Global  Positioning  System  III 


Program  Essentials 

Program  Performance  (fiscal  year  2012 

dollars  in  millions) 

Prime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Program  office:  B  Segundo,  CA 

0572008 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$2,524.2 

$2,694.8 

6.8 

R&D:  S924.6  million 

Procurement  cost 

$1,417.2 

$1,515.8 

7.0 

Procurement:  $1 ,435.0  million 

Total  program  cost 

$3,941.4 

$4,210.6 

6.8 

Total  funding:  $2,359.6  miHion 

Program  unit  cost 

$492,672 

$526,323 

6.8 

Procurement  quantity:  6 

Total  quantities 

8 

8 

0.0 

Acquisition  cycle  time  (months) 

NA 

NA 

NA 

we  could  not  caKuuee  acquisson  cycle  dme*  Ibr  me  drsr  mcremen  or  me  GPS  IH  piogrqn  oecause 

Initial  ooeraconai  cacaMty  wni  nor  occur  laiDi  eatenitee  rrom  a  tuture  mcfemeni  are  relded. 

Gray  Eagle  UAV 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  General  Atomics 

As  of 

Latest 

Percent 

Aeronautical  Systems.  Inc. 

(H^OOS 

08/2011 

change 

program  office:  Redstone  Arsenal,  AL 

Research  and  development  cost 

$344.9 

$946.2 

174.4 

Funding  needed  to  complete: 

Procurement  cost 

$670.4 

$3,400^ 

4072 

R&D:  $226.1  milkon 

Total  program  cost 

$1,015.2 

$5,158.9 

408.2 

Procurement:  $2,089.0  million 

Program  unit  cost 

$203,046 

$166,416 

-18.0 

Total  funding:  $3,006.9  miflion 

Totai  quantities 

5 

31 

520.0 

Procurement  quantity:  16 

Acquisition  cycie  time  (months) 

50 

TBD 

NA 

tctal  quantXM  mcmoe  31  plamon  cats  «im  4  aircraft  eacn  The  prooram  will  also  buy  21  alrcraq  lo 

iBp-ace  mose  lost  tnrougn  acnion  and  7  traninq  aircrad.  for  a  total  or  t  S2 

HC-130/MC  -130  Recapitalization  Program 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Program  office:  Wright-F^tterson  AFB, 

03/2010 

12/2010 

change 

OFi 

Research  and  development  cost 

$153.2 

$152.8 

-0.3 

Funding  needed  to  compiete: 

Procurement  cost 

$7,699.3 

$12,621.9 

63.9 

R&D:  $82.6  million 

Total  program  cost 

$8,364.2 

$13,090.8 

56.5 

Procurement:  $9,532.4  million 

program  unit  cost 

$113,029 

$107,302 

-5.1 

Total  funding:  $9,812.7  milion 

Total  quantities 

74 

122 

64.9 

Procurement  quantity:  91 

Acquisition  cycle  time  (months) 

NA 

NA 

NA 
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IDECOM  Block  4 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

F>rime  contractor  ITT  Electronic 

As  of 

Latest 

Percent 

Systems 

06/2008 

10/2011 

change 

Program  office:  Patuxent  River.  MD 

Research  and  development  cost 

$220.2 

$252.0 

14.5 

Funding  needed  to  complete: 

Procurement  cost 

$474.2 

$569.5 

20.1 

R&D:  $121.4  rrxINon 

Total  program  cost 

$694.4 

$821.5 

18.3 

Procurement:  $569.5  million 

FYogram  unit  cost 

$4,340 

$4,324 

-0.4 

Total  funding:  $690.9  miHion 

Total  quantities 

160 

190 

18.8 

Procurement  quantity:  190 

Acquisition  cycle  time  (months) 

59 

54 

-8.5 

Joint  High-Speed  Vessel 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Austal  USA 

As  of 

Latest 

Percent 

Program  office:  Washington,  DC 

02/2009 

12/2010 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$128.4 

$138.0 

7.4 

R&D:  $23.0  million 

Procurement  cost 

$3,507.9 

$3,536.1 

0.8 

Procurement:  $2,202.8  million 

Total  program  cost 

$3,636.4 

$3,674.1 

1.0 

Total  funding:  $2,225.9  million 

Program  unit  cost 

$202,020 

$204,116 

1.0 

Procurement  quantity:  11 

Total  quantities 

18 

18 

0.0 

Acquisition  cycle  time  (months) 

48 

50 

4.2 

Joint  Land  Attack  Cruise  Missile  Defense 


Program  Essentials 
F*rime  contractor  Raytheon 
Program  office:  Redstone  Arsenal,  AL 
Funding  needed  to  complete: 

R&D:  $634.1  million 
Procurement:  $5,199.4  million 
Total  funding:  $5,948.7  million 
Procurement  quantity:  14 


Program  Performance  (fiscal  year  2012  dollars  in  millions) 

As  of  Latest  Percent 

08/2005  12/2010  change 

Research  and  development  cost 

$2,005.5 

$2,523.2 

25.8 

F>rocurement  cost 

$4,588.7 

$5,199.4 

13.3 

Total  program  cost 

$6,665.9 

$7,857.8 

17.9 

Program  unit  cost 

$416,619 

$491,112 

17.9 

Total  quantities 

16 

16 

0.0 

Acquisition  cycle  time  (months) 

97 

103 

6.2 
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Joint  Precision  Approach  and  Landing  System 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  Raytheon 

As  of 

Latest 

Percent 

Program  office:  Lexington  Park,  MD 

07/2008 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$792.1 

$753.5 

-4.9 

R&D:  $183.9  milteon 

Procurement  cost 

$213.2 

$222.7 

4.4 

Procurement:  $222.7  million 

Total  program  cost 

$1,012.3 

$983.3 

-2.9 

Total  tundirtg:  $406.6  million 

Program  unit  cost 

$27,359 

$26,575 

-2.9 

Procurement  quantity:  26 

Total  quantities 

37 

37 

0.0 

Acquisition  cycle  time  (months) 

75 

77 

2.7 

Airborne  and  Maritime  Joint  Tactical  Radio  System 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

F>rime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Program  office:  San  Diego.  CA 

10/2008 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$1,945.0 

$1,957.0 

0.6 

R&D:  $593.7  million 

FHocurement  cost 

$6,209.0 

$6,203.8 

-0.1 

Procurement:  $6,203.8  million 

Total  program  cost 

$8,154.1 

$8,160.8 

0.1 

Total  funding:  $6,797.5  million 

Program  unit  cost 

$.301 

$.301 

0.1 

Procurement  quantity:  26,878 

Total  quantities 

27.102 

27,102 

0.0 

Acquisition  cycle  time  (months) 

80 

91 

13.8 

me  program  onice  reported  guantcee  m  terms  of  civanneis  ratner  than  radios 

Joint  Tactical  Radio  System  Handheld 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  General  Dynamics  C4 

As  of 

Latest 

Percent 

Systems,  Inc. 

05/2004 

11/2011 

change 

Program  office:  San  Diego.  CA 

Research  and  development  cost 

$544.7 

$1,272.3 

133.6 

Funding  needed  to  complete: 

Procurement  cost 

$9,492.8 

$7,085.7 

-25.4 

R&D:  $352.5  nkllion 

Total  program  cost 

$10,037.5 

$8,357.9 

-16.7 

Procurement:  $7,022.1  million 

Program  unit  cost 

$.031 

$.031 

1.0 

Total  funding:  $7,374.6  million 

Total  quantities 

328,674 

270,951 

-17.6 

Procurement  quantity:  264,019 

Acquisition  cycle  time  (months) 

85 

104 

22.4 
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LHA  Replacement  Amphibious  Assault  Ship 


Program  Essentiais 

Program  Performance  (fiscal  year  2012 

dollars  in  millions) 

Prime  contractor  Huntirtgton  Ingalls 

As  of 

Latest 

Percent 

Industries 

01/2006 

12/2010 

change 

F*rogram  office:  Washington.  DC 

Research  and  development  cost 

$220.9 

$350.9 

58.8 

Funding  needed  to  compiete: 

Procurement  cost 

$2,959.2 

$9,742.8 

229.2 

R&D:  $97.3  million 

Total  program  cost 

$3,180.2 

$10,095.2 

217.4 

Procurement:  $5,627.9  million 

Program  unit  cost 

$3,180,150 

$3,365,053 

5.8 

Totai  funding:  $5,726.2  miilion 

Total  quantities 

1 

3 

200.0 

Procurement  quantity:  1 

Acquisition  cycie  time  (months) 

146 

165 

13.0 

Littoral  Combat  Ship 


Program  Essentials 

Program  Performance  (fiscal  year  2012 

dollars  in  millions) 

F*rime  contractor  Austai  USA,  Generai 

As  of 

Latest 

Percent 

Dynamics.  Lockheed  Martin 

05/2004 

12/2010 

change 

Program  office:  Washington,  DC 

Research  and  development  cost 

$887.0 

$3,520.1 

296.9 

Funding  needed  to  compiete: 

Procurement  cost 

$471.6 

$29,136.1 

6,078.2 

R&D:  $1,112.5  million 

Total  program  cost 

$1,358.6 

$32,867.8 

2,319.3 

Procurement:  $25,(X)1.1  million 

Program  unit  cost 

$339.6 

$597,596 

76.0 

Total  funding:  $26,325.2  million 

Total  quantities 

4 

55 

1,275.0 

Procurement  quantity:  47 

Acquisition  cycle  time  (months) 

41 

116 

182.9 

Ca6t  Oaa  are  ftir  me  eeatrarne  orty  Tl>e  2004  esHnate  corresponds  win  program  Innatlor  It  was 

pre-niieslone  B  arid  ad  not  rePect  me  run  £5-snp  program,  tesearcn  and  deveropment  runaing 

Includes  delai  design  and  construction  or  two  stups. 

Mobile  User  Objective  System 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Space  Systems 

09/2004 

08/2011 

change 

Program  office:  San  Diego.  CA 

Research  and  development  cost 

$3,647.7 

$4,218.3 

15.6 

Funding  needed  to  complete: 

Procurement  cost 

$3,035.0 

$2,694.3 

-11.2 

R&D:  $470.1  million 

Total  program  cost 

$6,721.3 

$6,978.2 

3.8 

Procurement:  $1,125.1  million 

Program  unit  cost 

$1,120,222 

$1,163,035 

3.8 

Total  funrfing:  $1,595.2  million 

Total  quantities 

6 

6 

0.0 

Procurement  quantity:  1 

Acquisition  cycie  time  (months) 

90 

116 

28.9 

Tne  latss:  cost  data  do  not  reiiect  m«  current  cost  or  me  program.  A  new  acgmsioon  program  oaseilne 

ri3s  rxx  yet  Deen  approved. 
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MQ-4C  BAMS  UAV 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prwne  contractor  Northrop  Grumman 

As  of 

Latest 

Percent 

Systems  Corporation 

02/2009 

12/2010 

change 

Program  office:  Patuxent  River,  MD 

Research  and  development  cost 

$3,141.7 

$3,245.6 

3.3 

Funding  needed  to  complete: 

FTocurement  cost 

$9,323.4 

$9,413.9 

1.0 

R&D:  $1,657.1  million 

Total  program  cost 

$12,847.6 

$13,052.4 

1.6 

Procurement:  $9,413.9  million 

FTogram  unit  cost 

$183,537 

$186,463 

1.6 

Total  funding:  $11,422.2  million 

Total  quantities 

70 

70 

0.0 

Procurement  quantity:  65 

Acquisition  cycle  time  (months) 

92 

92 

0.0 

Navy  Multi-band  Terminal 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Raytheon 

As  of 

Latest 

Percent 

Program  office:  San  Diego.  CA 

12/2006 

08/2011 

change 

Funding  needed  to  complete: 

Research  arxl  development  cost 

$697.2 

$666.2 

-4.4 

R&D:  $41.1  million 

Procurement  cost 

$1,623.7 

$1,214.4 

-252 

Procurement:  $992.6  million 

Total  program  crjst 

$2,321.0 

$1,880.7 

-19.0 

Total  funding:  $1,033.8  milion 

program  unit  cost 

$6,970 

$6,186 

-11.2 

Procurement  quantity:  189 

Total  quantities 

333 

304 

-8.7 

Acquisition  cycle  time  (months) 

107 

107 

0.0 

P-8A  Poseidon 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

FTirtre  contractor  Boeing 

As  of 

Latest 

Percent 

Program  office:  Patuxent  River,  MD 

05/2004 

08/2011 

change 

Funding  needed  to  complete: 

Research  arxl  development  cost 

$7,531.5 

$8,215.3 

9.1 

R&D:  $1,232.4  million 

Procurement  cost 

$23,365.2 

$24,157.2 

3.4 

Procurement:  $20,087.9  million 

Total  program  cost 

$31,034.3 

$32,969.3 

6.2 

Total  funding:  $21,839.0  milion 

Program  unit  cost 

$269,864 

$270,240 

0.1 

Procurement  quantity:  104 

Total  quantities 

115 

122 

6.1 

Acquisition  cycle  time  (months) 

160 

160 

0.0 

TO  0030,  RT  040 

Report  No.  SERC-2013-TR-040-2 
Revised  September  3,  201 3 

47 


Contract  Number:  H98230-08-D-01  71 


UNCLASSIFIED 


Reaper  UAV 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  General  Atomics 

As  of 

Latest 

Percent 

Aeronauticai  Systems,  Inc. 

02/2008 

0e.'2011 

change 

Program  office:  Wright-Patterson  AFB, 

Research  and  development  cost 

$420.1 

$920.3 

119.1 

OH 

Procurement  cost 

$2,111.5 

$10,848.3 

413.8 

Fundirtg  needed  to  complete: 

Total  program  cost 

$2,637.1 

$11,918.7 

352.0 

R&D:  $420.5  million 

Program  unit  cost 

$25,115 

$29,871 

18.9 

Procurement:  $7,962.6  million 

Total  quantities 

105 

399 

280.0 

Total  funding:  $8,473.5  million 

Acquisition  cycle  time  (months) 

79 

94 

19.0 

Procurement  quantity:  240 

Space-based  Infrared  System  Program 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Pnme  contractor  Lockheed  Martm 

As  of 

Latest 

Percent 

Program  office:  B  Segundo,  CA 

10/1996 

07/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$4,376.3 

$11,586.3 

164.7 

R&D:  $2,131.3  million 

Procurement  cost 

$0.0 

$6,429.3 

NA 

Procurement;  $3,599.4  million 

Total  program  cost 

$4,596.5 

$18,266.7 

297.4 

Total  furrrtng;  $5,743.9  miflion 

Program  unit  cost 

$919,301 

$3,044,443 

231.2 

Procurement  quantity:  2 

Total  quantities 

5 

6 

20.0 

Acquisition  cycle  time  (months) 

86 

TBD 

NA 

TDe  199€  (UC3  mow  no  procurement  cost  as  ttte  Mr  Force  ptanneo  to  use  researcn  ana  oeveropment 

finds  IP  buy  all  fve  sateliitee.  Tne  cost  or  tbe  two  MEO  lepienistvnent  sensors  is  not  nauaed  in  emer 

column 

Standard  Missile  6  ERAM 


Program  Performance  (fiscal  year  2012  dollars  in  millions) 


Program  Essentials 

Prime  contractor  Raytheon  Missile 

Systems 

Program  office:  Arlington,  VA 
Funding  needed  to  complete: 

R4D:  S7.6  million 
Procurement:  S4, 808.9  million 
Total  furxfirrg:  $4,816.6  milion 
Procurement  quantity:  1,111 


Research  and  development  cost 
Procurement  cost 
Total  program  cost 
Program  unit  cost 
Total  quantities 

Acquisition  cycle  time  (months) 


As  of 

Latest 

Percent 

07/2004 

12/2010 

change 

$1,073.8 

$973.5 

-9.3 

$4,626.4 

$5,323J2 

15.1 

$5,700.2 

$6,296.7 

10.5 

$4,750 

$5,247 

10.5 

1,200 

1.200 

0.0 

75 

94 

25.3 
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IV.  Next  steps 


The  most  pressing  need  in  this  research  is  access  to  information  on  completed  programs  that  will  help 
characterize  the  connection  between  some  definitions  of  complexity  and  the  post  hoc 
prediction/realization  of  technical  risk. 


In  addition,  we  shall  pursue  more  case  studies,  more  literature,  and  more  methods  of  characterizing 
complexity  of  products  and  organizations,  including  interviewing  acknowledged  experts. 
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Appendix  A:  Literature  Review  on  System  Complexity  and  Risk 


This  section  provides  a  summary  and  synthesis  of  findings  from  a  survey  of  the  iiterature  on  system 
compiexity  and  cost,  scheduie  and  performance  risk  in  engineering  deveiopment  programs.  Severai 
reievant  and  exempiar  papers  discussed  in  detaii. 

The  survey  was  restricted  to  pubiicaiiy-accessibie,  freeiy-avaiiabie  documents  via  the  internet  found  by 
searching  on  combinations  of  the  keywords  "compiexity",  "compiex",  "compiicated",  "risk", 
"uncertainty",  "overrun",  "siip",  "shortfaii",  "cost",  "scheduie",  "performance",  "acquisition", 
"deveiopment",  "engineering",  "engineered",  "technicai",  "quantitative",  "indicators",  "factors", 
"moduiar",  "adaptive",  "adaptabie",  "system",  "modei",  "architecture",  "anaiysis",  and  "assessment". 
From  over  2,000  web  sites  visited,  approximateiy  600  articies  were  reviewed.  Roughiy  three-quarters 
focused  on  risk  in  system  acquisition,  with  some  reference  to  system  compiexity.  The  remaining  quarter 
focused  on  compiexity  of  engineered  systems,  with  some  reference  to  deveiopment. 


A.l  Summary  OF  Findings 

This  section  summarizes  both  what  was  found  in  the  iiterature,  and  what  was  notabie  by  its  absence. 

Many  papers  asserted  that  increased  compiexity  was  correiated  with  increased  deveiopment  time  and 
cost.  Data  and  evidence  supporting  this  intuitive  ciaim  was  sparse.  Most  of  the  papers  were  theoreticai, 
often  using  a  "toy"  system  modei  to  iiiustrate  the  methods,  but  did  not  provide  evidence  that  their 
compiexity  metrics 

(1)  couid  be  evaiuated  from  data  on  DoD  systems  avaiiabie  during  their  deveiopment, 

(2)  were  good  predictors  of  deveiopment  time  and  cost,  or 

(3)  were  predictors  of  cost  increase,  scheduie  siip  or  performance  shortfaii. 

Of  the  haif-dozen  articies  that  appiied  compiexity  metrics  to  acquisition  program  data  and  anaiyzed  the 
reiationship  to  cost  and  scheduie,  there  were  oniy  three  distinct  anaiyses.  A  series  of  papers  by  the 
same  authors  anaiyzed  from  different  perspectives  one  set  of  Major  Defense  Acquisition  Program 
(MDAP)  data  provided  by  OSD.  The  finai  report  by  the  Boeing  team  on  the  DARPA  META  ii  program 
anaiyzed  data  from  two  different  divisions  of  the  Boeing  Corporation.  A  PhD  thesis  out  of  the  Air  Force 
institute  of  Technoiogy  anaiyzed  an  aircraft  avionics  data  set.  These  papers  are  reviewed  in  detaii. 

The  papers  on  compiexity  addressed  the  compiexity  of  the  system  design,  but  did  not  specify  the 
appropriate  ievei  of  architecture  and  design  data  for  anaiysis.  This  begs  the  question  of  whether  the 
design  information  is  avaiiabie  eariy  enough  in  the  acquisition  process  to  guide  architecture  and  design 
decision.  The  papers  using  the  MDAP  data  provided  by  OSD,  used  Systems  Engineering  artifacts 
produced  at  Miiestone  B. 

Many  of  the  articies  on  acquisition  risk  identified  the  turbuience  in  the  system  requirements  and 
interdependencies  among  the  requirements  (either  antagonistic  or  synergistic)  as  sources  of  deiay,  cost 
increase,  time  and  cost  uncertainty,  it  is  possibie  that  compiexity  anaiysis  couid  be  appiied  to  the 
network  of  requirements  (or,  more  generaiiy  the  system  baseiine,  which  consists  of  the  capabiiity  needs, 
the  system  requirements,  system  functionai  decomposition  and  requirements),  and  to  the  change  in  the 
baseiine  over  time.  This  might  provide  usefui  and  timeiy  insights  into  acquisition  risk  and  compiexity  of 
the  system  baseiine.  These  data  are  avaiiabie  to  the  acquisition  Program  Manager's  Office,  are 
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developed  over  time,  and  could  potentially  benefit  from  feedback.  No  analyses  of  requirements  or 
system  baseline  complexity  and/or  change  in  complexity  were  found  in  the  literature. 

The  term  "complexity"  was  not  used  consistently  throughout  the  literature.  Many  of  the  papers, 
especially  those  focused  primarily  on  development  risk,  used  the  terms  "complex"  and  "complicated" 
interchangeably,  generally  meaning  "systems  with  lots  distinct  parts  and  lots  of  connections  among  the 
parts."  The  papers  focusing  primarily  on  the  complexity  of  engineered  systems  used  a  variety  of 
descriptions  and  definitions  of  complexity.  A  number  of  the  papers  distinguished  between  "structural 
complexity"  and  "dynamic  complexity". 

"Structural  complexity"  was  used  to  refer  to  complexity  in  the  architecture  of  the  system.  Structural 
complexity  was  commonly  based  on  a  graph  of  nodes  (processors)  and  links  (interfaces)  representing 
the  system  architecture.  Metrics  ranged  from  simple  counts  of  the  number  of  links  and  nodes  (similar  to 
the  measures  of  "complicated"  systems),  to  refinements  using  algebraic  network  analysis  of 
interconnectedness.  Some  of  the  approaches  distinguished  between  the  number  of  instances  of  a  type 
of  node  and  the  number  of  distinct  types  of  nodes.  Some  distinguished  between  uni-directional  and  bi¬ 
directional  interfaces. 

"Dynamic  complexity"  refers  to  the  behavior  or  response  of  the  system  (e.g.,  states  and  transitions). 

The  term  was  used  variously  to  refer  to  systems  exhibiting  adaptive  response  to  external  states,  non¬ 
linear  change  in  response  depending  on  internal  state,  adaptive  response  to  internal  states,  self¬ 
organization,  cascading  effects,  unexpected  responses,  or  "emergent  behavior".  The  dynamic 
complexity  metrics  require  a  model  of  the  system  behavior  and  response.  Many  of  the  papers 
discussing  dynamic  complexity  did  not  present  computational  metrics.  Some  papers  used  a  system  state 
transition  diagram  model  of  the  system  dynamics,  then  applied  analysis  methods  similar  to  those  used 
for  architecture  structure  graph  complexity. 

Some  of  the  papers  distinguished  between  observable  complexity  in  the  system  model,  and  hidden 
complexity  within  nodes  and  links.  Some  of  the  papers  distinguished  between  complexity  inherent  in 
the  system  design,  and  apparent  complexity  due  to  incomplete  models,  incomplete  analysis,  and 
incomplete  characterization  of  the  boundary  conditions.  These  distinctions  are  related  to  the 
unresolved  issue  of  selecting  the  granularity  or  level  of  resolution  of  the  system  description,  i.e.,  the 
level  or  scope  of  components  or  subsystem  to  represent  as  distinct  nodes.  There  is  a  tradeoff  between 
the  size  of  the  model,  and  risk  of  errors  in  the  model,  versus  errors  due  to  ignorance  of  the  behavior, 
response,  and  internal  states  of  nodes  and  links.  None  provided  guidelines  for  determining  the 
appropriate  level  of  granularity  for  analysis. 

State  transition  diagrams  to  analyze  dynamic  complexity  suffer  from  combinatorial  explosion.  A 
network  with  5  nodes  and  5  links,  where  each  node  and  link  has  two  possible  states  (e.g.,  busy  and  not 
busy,  or  operative  and  not  operative,  at  capacity  or  below  capacity),  has  1024  possible  states  (2  to  the 
10*^  power),  and  over  1,000,000  possible  state  transitions  (1024  squared  minus  1024).  Complexity 
analysis  requires  determining  which  states  the  system  can  actually  occupy,  and  which  transitions  it  can 
experience.  Reducing  the  level  of  granularity  to  limit  the  size  of  the  model  increases  the  "hidden" 
complexity. 

Most  of  the  complexity  metrics  relied  on  a  node-and-link  graph  representations  of  the  system  in  which 
nodes  perform  processes,  and  interfaces  exchange  data,  energy,  material,  physical  position,  etc. 
between  nodes.  Processors  and  interfaces  have  performance  properties  beyond  being  logical  nodes  and 
links  in  a  graph:  capacity,  latency,  noise,  losses,  etc.  When  there  is  sufficient  design  margin  (reserve 
capacity)  so  that  there  is  negligible  risk  that  the  component  or  interface  is  overloaded  or  non-operative. 
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then  it  may  be  possible  to  omit  the  node  or  link  and  its  internal  states  from  the  analysis.  When  there  is 
non-negligible  risk,  then  the  node  or  link  and  its  internal  states  should  be  included  in  the  analysis. 

An  important  class  of  interfaces  not  addressed  in  the  papers  is  insulators  and  isolators  whose  purpose  is 
to  prevent  exchange  of  force,  energy,  signals,  etc.  between  nodes  and  between  links  (e.g.,  prevent  cross¬ 
talk,  short-circuits,  vibration  transfer,  thermal  degradation,  etc.).  Failure  of  insulators  and  isolators,  or 
temporary  failure  when  they  reach  their  excursion  limit,  create  short  circuits  that  can  radically  change 
the  response  of  the  system. 

A  widely  used  approach  to  compute  graph  complexity  involved  the  "graph  energy",  computed  from  the 
eigenvalues  of  the  adjacency  matrix  representing  the  graph  of  either  the  architecture  structure  or  the 
state  transition  diagram.  Closed-form  calculation  of  the  graph  energy  is  only  possible  for  simple  graphs, 
e.g.,  trees  structures  without  loops  or  lattice  structures.  Research  on  robust  methods  to  estimate  the 
approximate  graph  energy  for  arbitrary  graphs  is  an  area  of  on-going  research. 

A  less  widely  adopted  approach  to  complexity  was  to  use  a  measure  of  the  information  content  in  a 
minimal,  irreducible,  specification  of  the  system  model. 

Axiomatic  Design  provides  an  alternative  view  of  complexity  in  terms  of  the  probability  that  the  system 
can  perform  the  functions  required  of  it  at  any  given  time.  Axiomatic  Design  focusses  on  the 
relationship  between  system  structure  and  functions.  In  principle,  it  addresses  the  frequency  and 
duration  of  functions,  and  multiple  simultaneous  functions. 

Axiomatic  Design  suggests  approaches  to  understand  the  modularity  inherent  in  a  design,  either  by 
modules  associated  with  overlapping  functions,  or  block-diagonal  modularization  to  minimize  interfaces 
between  blocks.  Several  papers  contained  analytic  approaches  or  metrics  incorporating  modularity  into 
the  complexity  metrics.  There  is  a  small  body  of  literature  on  multi-scale  complexity,  but  it  is  oriented 
towards  biological  organisms,  not  engineered  systems. 


A.2  Reviews  of  Selected  Papers  and  Presentations 

This  section  contains  reviews  of  all  of  the  papers  and  presentations  analyzing  the  relationship  of 
complexity  to  cost  and  schedule  performance  (beginning  with  the  three  analyzing  the  OSD  MDAP  data 
set),  followed  by  reviews  of  selected  papers  exemplify  major  issues  and  approaches  in  complexity 
metrics  for  engineered  systems.  The  reviews  are  organized  under  the  following  topics: 

•  Systems  complexity  and  development  risk 

•  Structural  and  dynamic  complexity  metrics  from  graph  complexity 

•  Modularity  considerations  and  metrics  in  system  complexity 

•  Axiomatic  Design  approach  to  complexity 

•  Functional  and  contextual  complexity 

•  Apparent  complexity  in  flight  software 

•  Adaptability  metrics  to  measure  complexity 

•  Aspects  of  complexity  in  design 
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A,2,l  System  Complexity  and  Development  Risk 

Programmatic  and  Constructive  Interdependence:  Emerpinq  Insights  and  Predictive  Indicators  of 

Development  Resource  Demand,  M.  Kasunic,  M.M.  Brown,  PI.  Hardin,  J.M.  McCurley,  2010 

http://repositorv.cmu.edu/cgi/viewcontent.cgi?article=1008&context=sei 

Kasunic  et  al  [1]  describe  a  series  of  research  efforts  investigating  the  role  of  interdependence  in  the 
acquisition  of  Major  Defense  Acquisition  Programs  (MDAPs).  The  research  initiative  was  sponsored  by 
the  Office  of  the  Secretary  of  Defense  (OSD).  The  overall  goal  of  the  research  was  to  identify,  quantify, 
and  assess  the  degree  of  programmatic  and  constructive  interdependence  and  to  assess  the  effects  of 
interdependence  on  program  risk.  The  report  summarizes  the  results  of  five  research  studies  that  were 
conducted  from  2004  to  2009. 

Study  1  explored  the  qualitative  factors  that  confound  program  cost  and  schedule  estimation.  The  study 
identified  specific  risk  indicators  related  to  requirements,  institutional  factors,  sustainment,  and  team 
performance. 

Study  2  employed  data-mining  and  statistical  analyses  to  determine  whether  Defense  Acquisition 
Executive  Summary  (DAES)  reports  and  Select  Acquisition  Reports  (SARs)  can  be  used  to  forecast 
program  performance.  An  interesting  result  from  this  study  is  that  there  was  no  evidence  that  such 
indicators  are  effective  in  predicting  program  breaches. 

Studies  3-5  employed  network  analysis  techniques  to  quantitatively  characterize  programmatic  and 
constructive  interdependencies  in  the  acquisition  enterprise.  These  last  three  studies  culminated  in 
graphical  models  that  relate  interdependence  and  program  cost.  The  research  study  found  no  evidence 
that  indicators  reported  within  DAES  reports  or  SARs  predict  program  breach  events. 

Simple  Parametric  Model  For  Estimating  Development  (RDT&E)  Costs  for  Large-Scale  Systems,  R.R.  Jones, 
P.  Hardin,  A.  Irvine,  2005 

http://www.technomics.net/files/downloads/papers/ISPASCEA0609-Parametric.pdf 

Jones,  Hardin  and  Irvine  [2]  analyzed  data  provided  by  OSD(AT&L).  The  sponsor  provided  data  on  21 
Major  Defense  Acquisition  Programs  (MDAPs).  The  data  included  the  initial  RDT&E  cost  estimates  from 
Selected  Acquisition  Reports  (SARs),  and  architecture  structure  metrics  calculated  from  DoDAF  SV-6 
architecture  views:  numbers  of  send-only  nodes,  receive-only  nodes,  send-and-receive  nodes,  all  nodes, 
one-way  links,  two-way  links,  and  all  links.  The  analysis  of  the  relationship  between  the  total  number  of 
nodes  and  the  total  number  of  links  showed  a  strong  linear  correlation,  with  one  noticeable  outlier.  The 
analysis  initial  RDT&E  cost  showed  a  strong  correlation  with  the  square  of  the  number  of  links.  The  data 
clustered  into  three  groups  (a  large  number  of  data  points  with  low  cost  and  low  number  of  links,  three 
points  with  mid-range  cost  and  number  of  links,  and  one  point  with  high  cost  and  number  of  links),  so 
from  a  statistical  analysis  view,  there  were  only  three  distinct  data  points.  The  authors  found  a 
complicated  non-linear  formula  relating  cost  to  the  architecture  structure  metrics  with  almost  perfect 
correlation.  However  the  number  of  implicit  and  explicit  parameter  exceeded  the  statistically  significant 
degrees  of  freedom  in  the  data  set  as  analyzed.  Regardless  of  the  statistical  details,  the  report  presents 
system  architecture  metrics  derived  from  required  artifacts  (DoDAF  SV-6  architecture  is  required  prior  to 
Milestone  B),  with  strong  correlation  to  RDT&E  cost. 

The  sponsoring  agency  was  kind  enough  to  provide  the  SERC  with  the  original  data,  updated  to  include 
the  RDT&E  cost  estimates  as  of  2008.  We  re-analyzed  the  data,  with  care  not  to  "over-fit."  We 
conducted  a  bootstrap  analysis  (replicated  random  partitions  of  the  data  into  "training/calibration"  and 
"test/evaluation"  data  sets).  Analysis  in  log-log  space  showed  a  strong  linear  correlation  between  the 
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initial  RDT&E  cost  estimates  and  the  number  of  links  in  the  DoDAF  SV-6  diagrams.  In  log-log  space,  the 
RDT&E  cost  estimates  and  the  number  of  links  were  nicely  distributed  over  their  range.  The  dispersion 
about  the  linear  fit  provided  an  estimate  in  the  uncertainty  (error)  between  initial  RDT&E  estimates  and 
predictions  from  the  number  of  links.  These  results  showed  that  70-percent  of  the  variance  in  the 
logarithm  of  the  initial  estimates  of  RDT&E  cost  in  one  data  set  was  predicted  by  (a)  the  logarithm  of  the 
number  of  links,  and  (b)  the  linear  relationship  between  the  logarithm  of  initial  RDT&E  cost  estimates 
and  logarithm  of  the  number  of  links  derived  from  a  sequestered  data  set. 

No  relationship  in  the  change  in  RDT&E  costs  from  the  initial  estimates  and  the  2008  RDT&E  cost 
estimate  bore  any  relationship  to  the  architecture  parameters.  Since  the  programs  were  started  at 
different  times  and  on  different  schedules,  the  programs  developments  from  inception  to  2008  were 
not  samples  from  the  same  population.  The  DoDAF  SV-6  diagrams  and  architecture  data  were  not 
updated. 

Programmatic  Complexity  &  Interdependence:  Emergina  Insights  and  Predictive  Indicators  of 

Development  Resource  Demand,  R.  Flowe,  M.  Brown,  P.L  Hardin,  2000 

http://acquisitionresearch.net/  files/FY2009/NPS-AM-09-058.pdf 

Flowe,  Brown  and  Flardin  [3]  prepared  report  for  the  Defense  Science  Board  addressing  the  effects  of 
technical  interdependence  among  programmatically-independent  acquisitions,  whether  explicit  as  in 
Systems-of-Systems,  or  implicit.  The  report  finds  that  interdependencies  have  non-linear  scaling  effects 
that  are  not  captured  in  technical  development  and  integration  cost  estimates.  The  research  used  data 
extracted  from  formal  program  documentation  including  Defense  Acquisition  Executive  Summary 
(DAES)  Charts,  Selected  Acquisition  Reports  (SARs),  Budget  Item  Justification  Exhibits,  Information 
Support  Plans  (ISPs),  etc.  The  research  analyzed  dependencies  of  MDAP  programs  on  other  MDAP 
programs  and  on  non-MDAP  programs  that  were  not  required  to  report  program  status.  The  report 
presented  a  network  diagram  program  interdependence  and  cost  growth  of  the  MDAPs,  but  did  not 
quantitatively  analyze  the  data. 

The  network  consisted  of  21  MDAPs,  162  non-MDAP  programs,  10  direct  dependencies  between 
MDAPS,  and  257  dependencies  of  MDAPs  on  non-MDAP  programs.  On  average,  an  MDAP  program  had 
13  external  dependencies  (one  to  another  MDAP  program,  and  12  to  non-MDAP  programs).  78-percent 
of  the  non-MDAP  programs  had  exactly  one  dependent  MDAP;  the  remaining  22-percent  had,  on 
average,  2.6  dependent  MDAPs. 

The  fourteen  MDAPs  with  less  than  50%  cost  growth  had  an  average  of  11  external  dependencies 
(sample  standard  deviation  of  4),  while  the  seven  MDAPs  with  more  than  50%  cost  growth  had  an 
average  of  17  external  dependencies  (sample  standard  deviation  of  7).  The  difference  suggests  that 
more  external  dependencies  was  correlated  with  greater  cost  overrun,  but  the  statistical  confidence  is 
low  due  to  the  large  variance. 

Impact  of  weapon  system  complexity  on  systems  acquisition,  R.A.  Dietrick,  2006 

http://dtlweb.au.af.miI///exlibris/dtl/d3  1/apache  media/L2V4bGlicmlzL2R0bC9kM18xL2FwYWNoZV9t 

ZWRpYS81MDk3Nw==.pdf 

In  his  PhD  thesis,  Dietrick  [4]  used  the  number  interactions  among  components  as  the  theoretical 
complexity  metric.  Interactions  include  space,  energy,  information,  and  material  exchange.  Due  to  the 
difficulty  in  counting  the  actual  interactions  among  system  components,  as  a  practical  metric  the  paper 
uses  the  theoretical  upper  bound  on  the  number  of  interactions  among  components  as  the  practical 
measure.  The  upper  bound  is  N(N-l)/2  where  N  is  the  number  of  components.  The  paper 
acknowledges  that  the  level  of  resolution  to  identify  components  will  affect  the  results,  and  comparison 
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across  systems  required  analysis  at  the  same  level  of  resolution.  It  does  not  consider  modularity  and 
hierarchical  organization. 

The  thesis  presents  empirical  data  for  USAF  aircraft  showing:  (1)  increase  of  complexity  with  increase  of 
year  of  operating  capability,  (2)  increase  of  development  time  with  year  of  operating  capability,  and  (3) 
increase  of  development  cost  with  increase  of  development  time.  The  thesis  does  not  directly  analyze 
system  development  time  or  cost  as  function  of  complexity  on  an  individual  system  basis.  The  thesis 
provides  an  analysis  of  trends,  not  an  analysis  of  systems.  The  paper  does  not  address  cost  increase, 
schedule  slip,  or  performance  shortfall. 

META  II  Complexity  and  Adaptability  Final  Report,  D.  Stuart,  R.  Mattikalli,  D.  DeLaurentis,  J.  Shah,  2011 

http://www.darpa.mil/uploadedFiles/Content/Our  Work/TTO/Programs/AVM/Boeing%20META%20Fin 

al%20Report.pdf 

The  Boeing  team's  final  report  on  complexity  and  adaptability  metrics  for  the  DARPA  META  program 
(Stuart  et  al  [5])  took  an  investigative,  opportunistic  and  integrated  approach.  They  did  not  pursue  a 
sequence  of  first  developing  a  theory  or  computational  model  of  complexity,  then  seeking  data  to 
evaluate  metrics,  then  seeking  data  on  cost  and  schedule,  then  testing  the  ability  of  the  model/metric  to 
explain  the  variance  in  cost  and  schedule.  Instead,  the  team  identified  28  already-defined  factors  with 
potential  value  in  a  complexity  metric,  that  could  be  evaluated  from  available  system  architecture  data. 
Calculation  methods  for  each  of  the  inputs  are  included  in  the  report.  The  identified  two  sources  of 
aircraft  program  data  (Boeing  Commercial  Aircraft,  BCA,  and  Boeing  Defense  Systems,  BDS)  for  which 
system  architecture  data,  cost  data  and  schedule  data  were  available  (peak  labor  was  used  as  a  proxy 
for  cost).  Armed  with  this  data,  the  team  conducted  a  "combinatorial  search"  for  combinations  of  input 
terms  models  to  explain  the  variances  in  peak  labor  and  in  schedule,  using  regression  to  estimate  the 
coefficients,  including  linear  and  logarithm  operations  on  the  inputs,  additive  and  multiplicative 
relationships.  The  models  that  were  finally  selected  contained  up  to  six  terms. 

The  report  noted  that  significant  manual  effort  was  required  to  extract  the  architecture  data  for  the  BDS 
programs,  including  interviews  with  the  chief  engineers.  The  BCS  data  were  extracted  from  an 
automated  project  management  system.  The  sample  size  was  approximately  15  BCA  projects  and  15 
BCD  projects.  The  modeling  was  conducted  separately  for  the  BCA  and  BCD  projects,  presumably 
because  the  team  suspected  differences  due  to  the  type  of  project  and  data  sources.  Not  only  did  the 
coefficients  for  the  models  differ  between  the  two  data  sets,  but  different  inputs  and  functional  forms 
ended  up  being  selected. 

"Combinatorial  search"  modeling  is  at  risk  of  using  up  the  degrees  of  freedom  in  the  data,  unless  the 
sample  size  is  large.  There  were  28  inputs,  with  the  option  of  taking  logarithm  or  squaring  for  each,  and 
the  options  of  mixed  addition  and  multiplication,  there  were  many  more  possible  models  than  the 
sample  size  supported.  The  report  acknowledges  the  sample  size  issue. 

The  team  could  have  used  bootstrap  or  similar  techniques  to  examine  the  stability  and  validity  of  the 
models.  They  did  not  randomly  divide  the  data  into  a  pair  of  disjoint  groups,  apply  the  process  to  the 
different  groups  to  test  if  the  same  input  parameters  were  selected  for  both  groups.  Ideally,  in  this 
model  development  approach,  iterative  replication  of  the  following  steps  are  used  to  develop  and  justify 
the  model:  (1)  divide  the  population  into  three  groups,  (2)  use  the  first  group  to  select  the  input  terms 
and  non-linear  functions  of  the  model,  (3)  use  the  second  group  to  estimate  the  values  of  the 
coefficients,  and  (4)  use  the  third  group  to  evaluate  the  explanatory  power  of  the  model.  This  process  is 
repeated  with  different  random  partitions  to  analyze  the  stability  and  validity  of  the  modeling  results. 
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A.2.2  Structural  and  Dynamic  Complexity  Metrics  from  Graph  Complexity 

Meta  II  Complex  Systems  Design  and  Analysis  (CODA)  Final  Report,  B.  T.  Murray,  A.  Pinto,  R.  Skelding,  O. 
de  Week,  H.  Zhu,  S.  Nair,  N.  Shougarian,  K.  Sinha,  S.  Bopardikar,  and  L.  Zeidner,  2011 

http://www.dtic.mil/dtic/tr/fulltext/u2/a552676.pdf 

Murray  et  al  [6]  reported  on  the  United  Technologies  team  results  on  the  DARPA  META  II  program.  The 
report  covers  all  aspects  of  the  United  Technologies  effort  on  the  program.  Section  3.16,  "Complexity 
and  Adaptability  Metrics  in  Design"  is  particularly  relevant.  The  project  did  not  apply  the  methods  to 
real  systems  or  attempt  to  investigate  their  ability  to  explain  cost,  schedule,  overruns  and  slippage.  This 
paper  was  selected  to  review  as  an  exemplar  of  the  architecture  graph  analysis  methods. 

The  technical  approach  used  a  "graph"  model  of  the  system,  i.e.,  a  model  consisting  of  nodes  and 
interfaces  (information,  energy,  force,  momentum,  data,  signals,  fluids,  positional  relationship,  etc. 
exchanged  between  nodes).  For  computational  purposes,  the  graph  is  represented  by  the  Design 
Structure  Matrix  (DSM):  one  row  and  column  for  each  node,  a  one  in  the  matrix  if  there  is  an  interface 
from  the  row  node  to  the  column  node,  zero  otherwise,  and  zero  on  the  diagonal.  A  simplified,  non- 
directional  versions  of  the  DSM  is  the  association  matrix:  one  row  and  column  for  each  node,  a  one  in 
the  matrix  if  there  is  an  interface  in  either  direction  between  the  row  node  to  the  column  node,  zero 
otherwise,  and  zero  on  the  diagonal. 

Structural  complexity  refers  to  the  complexity  of  connections  between  subsystems.  The  proposed 
metric  was  the  number  of  components  plus  the  product  of  the  number  of  interfaces  times  the  "Graph 
Energy".  Graph  Energy  is  computed  from  the  adjacency  matrix,  a  non-directional  simplification  of  the 
DSM,  has  a  one  in  each  cell  if  the  row  node  and  column  node  have  an  interface  in  either  direction,  and 
zero  otherwise.  For  simple  graphs,  i.e.,  tree  structures  without  loops,  the  Graph  Energy  is  computed  as 
the  sum  of  the  absolute  values  of  the  eigenvalues  of  the  adjacency  matrix.  In  very  simple  geometries, 
loops  can  be  isolated  are  treated  as  a  single  node.  In  complex  geometries,  e.g.,  multiple  input  and 
output  nodes  within  loops,  nested  loops,  intersecting  loops,  etc.  other  computational  means  are 
needed,  such  as  Gibbs  sampling,  the  elimination  algorithm,  and  belief  propagation.  These  methods  are 
approximate  and  not  guaranteed  to  converge. 

Dynamic  complexity  refers  to  the  complexity  of  connections  between  transient  states  of  the  system. 
Instead  of  analyzing  the  links  between  nodes  of  the  system  architecture,  dynamic  entropy  examines 
links  between  states  of  the  system.  The  state  of  the  system  is  a  vector  with  an  element  for  each  node 
and  link.  Two  states  are  connected  if  there  is  a  single  event  that  will  cause  the  system  to  transition  from 
one  state  to  the  other.  Dynamic  complexity  is  computed  in  the  same  manner  as  structural  complexity, 
except  applied  to  the  state  association  matrix:  the  number  of  states  plus  the  number  of  state  transitions 
times  the  graph  energy  of  the  state  association  matrix. 

The  SVD  calculation  of  graph  energy  works  only  for  simple  graphs,  i.e.,  systems  without  feedback  loops. 
Other  methods  computational  methods  are  needed  to  estimate  the  graph  energy  for  arbitrary  graphs, 
and  specifically  for  systems  with  nested  and  intersecting  feedback  loops. 


A,2,3  Modularity  Considerations  and  Metrics  in  System  Complexity 

Degree  of  Modularity  in  Engineering  Systems  and  Products  with  Technical  and  Business  Constraints,  K. 

Hoitta-Otto  and  O.  de  Week,  2007 

http://strategic.mit.edu/docs/2  19  CERA  15  2  113.pdf 
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Hoitta-Otto  and  de  Week  [7]  present  a  structural  modularity  metric  and  "packing  factor"  metric  to 
complement  structural  complexity  metrics  (such  as  the  number  of  nodes  plus  the  number  of  links  times 
the  graph  energy).  The  graph  energy  measures  the  total  system  connectivity.  The  goal  of  the 
modularity  index  is  to  measure  the  concentration  of  connectivity.  The  idea  behind  the  modularity  index 
is  that  important  information  describing  system  connectivity  is  concentrated  in  the  subset  of 
components  that  are  highly  connected  across  the  network.  The  modularity  index  is  an  attempt  to 
measure  connectivity  concentration. 

The  structural  modularity  metric  is  derived  from  the  SVD  of  the  adjacency  metric,  as  is  the  graph  energy. 
The  graph  energy  is  the  sum  of  the  absolution  values  in  the  SVD.  Since  the  SVD  is  a  diagonal  matrix,  it 
can  be  collapsed  to  a  vector,  and  sorted  in  decreasing  order  of  absolute  value.  The  authors  use 
exponential  decay  as  a  model  of  the  decrease  in  sorted  absolute  value  SVD  elements.  The  structural 
modularity  metric  is  derived  from  the  estimated  rate  of  decay.  Other  measures  of  concentration  that  do 
not  assume  an  exponential  decay  could  be  formulated  to  measure  the  concentration  of  magnitudes  in 
the  SVD.  The  authors  show  how  the  modularity  in  structural  modularity  metric  dex  discriminates  among 
several  architectures  with  equal  numbers  of  nodes,  links  and  graph  energy. 

Using  the  SVD  to  compute  the  structural  modularity  metric  has  the  same  drawback  as  using  the  SVD  to 
compute  graph  energy:  it  only  works  for  "simple"  graphs  -  tree  structures  without  loops.  The  authors 
do  not  propose  an  approach  to  compute  the  metric  for  arbitrary  graphs. 

The  authors  also  propose  "packing  density"  as  a  modularity  metric  to  complement  the  structural 
modularity  metric.  The  packing  density  metric  is  the  ratio  of  the  sum  of  the  volume  of  space  needed  for 
each  component  to  operate  (including  space  to  move,  as  appropriate)  divided  by  the  total  volume  of  the 
system.  A  similar  metric  could  be  generated  for  weight  and  other  cumulative  constraints.  The  packing 
density  metric  does  not  account  for  systems  that  share  space  (one  moves  out  as  the  other  moves  in; 
since  the  motion  is  coordinated,  presumably  these  are  considered  to  be  one  component). 


A,2,4  Axiomatic  Design  Approach  to  Complexity 

Complexity  Theory  in  Axiomatic  Design,  T.  Lee,  2003 

http://citeseerx.ist.  psu.edu/viewdoc/download?doi=10. 1.1. 135. 4528&rep=repl&tvpe=pdf 

Lee's  PhD  thesis  [8]  was  selected  for  review  because  it  presents  a  principled  approach  that  expands  the 
notion  of  complexity  as  a  function  of  the  structure  and  dynamics  of  a  system  to  include  the  functions  of 
the  system. 


The  complexity  concept  in  axiomatic  design  theory  is  defined  as  a  measure  of  the  likelihood  of  not 
achieving  a  desired  set  of  functional  requirements.  In  this  thesis,  four  different  types  of  complexity  are 
identified  in  axiomatic  design  complexity  theory:  time-independent  real  complexity,  time-independent 
imaginary  complexity,  time-dependent  combinatorial  complexity  and  time-dependent  periodic 
complexity.  Time-independent  real  complexity  is  equivalent  to  the  information  content,  which  is  a 
measure  of  a  probability  of  achieving  functional  requirements.  Time-independent  imaginary  complexity 
is  defined  as  the  uncertainty  due  to  ignorance  of  the  interactions  between  functional  requirements  and 
design  parameters.  Time-dependent  complexity  consists  of  combinatorial  complexity  and  periodic 
complexity,  depending  on  whether  the  uncertainty  increases  indefinitely  or  occasionally  stops  increasing 
at  certain  point  and  returns  to  the  initial  level  of  uncertainty.  In  this  thesis,  existing  definitions  for  each 
of  the  types  of  complexity  are  further  elaborated  with  a  focus  on  time-dependent  complexity.  In 
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particular,  time-dependent  complexity  is  clearly  defined  using  the  concepts  of  time-varying  system 
ranges  and  time-dependent  sets  of  functional  requirements. 


The  Axiomatic  Design  model  is  that  a  Design  Matrix  (DM)  specifies  how  the  Design  Parameters  (DP)  are 
related  to  the  Functional  Requirements  (FR).  The  FR  are  input  to  the  design  process,  the  DP  are  output. 
The  axioms  (or  principles)  of  Axiomatic  Design  are: 

•  Independence  Axiom:  Maintain  the  independence  of  functional  requirements 

•  Information  Axiom:  Minimize  the  information  content. 

Information  content  is  defined  as  the  negative  probability  of  achieving  the  functional  requirements  over 
the  range  of  conditions.  The  paper  defines  real  complexity  as  sensitivity  of  the  information  content  to 
changes  in  Functional  Requirements,  and  imaginary  complexity  as  uncertainty  in  DP  values,  due  to 
uncertainty  in  FR  and  DM. 


A.2.5  Functional  and  Contextual  Complexity 

The  mathematics  of  IT  simplification,  R.  Sessions,  2011 

https://dl.dropboxusercontent.com/u/97323460/WebDocuments/WhitePapers/MathOflTSimplification- 

103.pdf 

Sessions  [9]  addresses  functional  complexity  (as  opposed  to  structural  or  dynamic  complexity).  The 
paper  suggests  that  functional  complexity  has  two  complementary  components:  internal  functional 
complexity  and  external  coordination  complexity.  The  goal  of  the  paper  is  to  develop  principles  to 
partition  a  system  to  simplify  development.  The  approach  is  inherently  hierarchical  and  can  be  applied 
to  hierarchical  partitioning  or  embedding  systems.  The  paper  does  not  explicitly  relate  the  metrics  to 
development  risk. 

In  principle  the  paper  takes  two  views  of  the  system:  as  a  stand-alone  system,  and  as  an  integral 
component  of  a  system-of-systems.  Internal  functional  complexity  is  a  measure  of  complexity  as  a 
stand-alone  system.  External  coordination  complexity  is  a  measure  of  complexity  of  the  role  in  a  system 
of  systems.  The  fundamental  difference  is  the  perspective,  not  the  computational  method.  Analogous 
methods  are  applied  in  both  perspectives. 

The  approach  computes  complexity  as  the  number  of  interactions  of  states  of  a  system  over  the  system 
functions.  The  number  of  states  of  a  system  computed  from  the  number  of  variables  (elements;  nodes), 
the  number  of  possible  states  for  each  node,  and  the  interdependencies  among  the  nodes  to  accomplish 
system  functions.  For  a  single  function,  it  disregards  all  nodes  whose  state  does  not  affect  the  function. 

It  determines  independent  groupings  according  to  the  rule  that  two  nodes  are  in  the  same  group  if  the 
performance  of  the  function  is  a  non-linear  function  of  the  two  nodes.  Within  a  grouping,  the  number 
of  states  is  the  product  of  the  number  of  states  pertaining  to  that  function  over  all  nodes  in  the  group. 
Different  functions  may  produce  some  of  the  same  groups.  The  measure  of  complexity  is  the  sum  over 
all  groups. 

Further  rationalization  in  defining  groupings  may  be  possible  (using  the  framework  of  normal  forms  in 
rational  database).  The  approach  does  not  analyze  overlap  between  functions.  Consider  two  groupings 
A  and  B  based  on  two  different  functions.  If  A  and  B  are  disjoint,  the  complexity  of  the  union  is  equal  to 
the  complexity  of  the  sum  (the  formulation  in  the  paper).  If  the  nodes  A  and  B  are  identical,  the 
combined  complexity  should  be  equal  distinct  states  over  both  function,  i.e.,  the  sum  of  the  individual 
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complexity  minus  the  number  of  overlapping  states.  If  A  and  B  partially  intersect,  the  complexity  is  the 
sum  of  the  individual  complexity  minus  the  number  of  overlapping  states. 


A.2.6  Apparent  Complexity  IN  Flight  Software 

NASA  Study  on  Flight  Software  Complexity,  D.L  Dvorak,  2009 
http://www.nasa.gov/pdf/418878main  FSWC  Final  Report.pdf 

The  NASA  study  on  flight  software  complexity  [10]  was  driven  by  a  perception  that  flight  software  was  a 
major  source  of  cost  and  time  growth.  The  goal  of  the  study  was  to  identify  the  problems  and  find  ways 
to  mitigate  those  problems.  In  this  sense,  the  study  was  an  empirical,  investigative  study,  not  a 
theoretical  study.  It  did  not  develop  a  formal  complexity  metric,  but  instead  develop  a  list  of  potential 
causes  and  indicators  to  track.  The  study  identified  a  handful  of  software  complexity  metrics  in  the 
literature  (e.g.,  cyclomatic  complexity,  Halstead  complexity,  Henry  and  Kafura  metrics,  Bowles  metrics. 
Trot  and  Zweben  metrics,  Ligier  metrics),  but  did  not  devote  resources  to  the  study  of  complexity 
metrics  because  the  issues  they  identified  were  not  well  addressed  by  the  metrics. 


The  study  adopted  the  IEEE  Standard  Computer  Dictionary  definition  of  'complexity'  as  "the  degree  to 
which  a  system  or  component  has  a  design  or  implementation  that  is  difficult  to  understand  and  verify" 
The  phrase  "difficult  to  understand"  indicates  that  complexity  is  inherently  about  the  knowledge  and 
understanding  of  the  personnel  involved  in  the  project.  Complexity  in  this  sense  is  not  a  computable 
property  of  the  engineered  system.  But  this  sense  of  complexity  is  directly  related  to  the  likelihood  of 
inaccurate  estimates  and  mistaken  decisions. 


Flight  software  complexity  is  related  to: 

•  How  difficult  it  is  for  a  programmer  to  implement  the  requirements  the  code  must  satisfy? 

•  How  difficult  it  is  for  a  tester  to  verify  that  the  code  satisfies  the  requirements  and  operates  in  an 
error-free  fashion? 

•  How  difficult  it  is  for  a  lead  developer  to  manage  the  development  of  the  flight  software  within  cost 
and  schedule? 

•  How  difficult  it  is  for  a  flight  software  maintenance  programmer  to  understand  the  original 
programmer's  work  if  the  software  must  be  modified  after  launch? 

•  How  difficult  it  is  for  a  new  programmer  on  a  later  mission  to  adapt  the  original  flight  software  as 
heritage  for  the  new  mission? 

•  How  difficult  it  is  to  predict  the  flight  software's  behavior,  which  in  turn  can  drive  much  more 
extensive  testing  and  more  operational  "hand-holding"  along  with  their  associated  higher  labor  costs? 


Factors  used  to  measure  a  flight  software  system's  essential  complexity  include: 

•  How  many  functions  the  flight  software  must  execute  and  monitor? 

•  How  many  hardware  components  the  flight  software  must  monitor,  command,  control,  and  query  for 
information? 
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•  How  many  connections  (both  hardware  and  software)  between  components  the  flight  software  must 
monitor  and  manage? 

•  How  many  control  modes  must  be  managed  and  executed? 

•  How  many  software  modules  must  be  implemented  in  order  to  satisfy  the  flight  software's 
requirements? 

•  How  much  coupling  there  is  between  software  modules? 

•  How  intricate/convoluted  the  code  is  within  a  module  (assuming  best  programming  practices,  this  is  a 
measure  of  the  complexity  of  an  associated  requirement  or  algorithm  itself)? 

•  How  many  tests  must  be  created  and  executed  in  order  to  verify  that  the  flight  software  has  satisfied 
its  requirements  and,  in  fact,  whether  it  is  even  possible  given  limited  time  and  cost  to  verify  satisfaction 
of  those  requirements  under  all  likely  scenarios? 

•  How  "state-of-the-art"  the  requirement  is  (reflected  in  how  demanding  performance  and  accuracy 
requirements  are  relative  to  contemporary,  heritage  systems)? 


A,2,7  Adaptability  Metrics  to  Measure  Complexity 

Desipninp  Systems  for  Adaptability  by  Means  of  Architecture  Options,  A.  Engel  and  T.  R.  Browning,  2006 
http://www.incose.org/symp2008/dmdocuments/paper  example01.pdf 

In  [11],  Engel  and  Browning  assert  that  the  objectives  of  design  for  adaptability  are  to  make  complexity 
manageable,  to  enable  parallel  work,  to  enable  efficient  recovery  from  mistakes,  and  to  accommodate 
future  uncertainty.  Adaptability  mitigates  against  the  risk  of  changing  needs  and  technologies.  In  the 
sense  of  the  IEEE  Standard  Computer  Dictionary  definition  of  'complexity'  as  "the  degree  to  which  a 
system  or  component  has  a  design  or  implementation  that  is  difficult  to  understand  and  verify", 
adaptability  is  the  antithesis  of  complexity. 

The  paper  by  Engel  and  Browning  []  develops  static  and  dynamic  approaches  to  evaluate  adaptability.  It 
develops  metrics  for  six  dimensions  of  adaptability  (functionality,  reliability,  usability,  efficiency, 
maintainability,  and  portability).  Each  of  these  dimensions  is  further  subdivided.  The  paper  applies  real 
options  theory  to  assess  value  of  a  design  including  the  present  value  of  the  future  options  the  design 
provides. 


A,2,8  Aspects  of  Complexity  in  Design 

Complexity  models  in  design,  C.  Earl,  C.  Eckert,  andJ.  Johnson,  2004 
http://oro.open.ac.Uk/7420/l/Complexitv  Models  2004.pdf 

Earl,  Eckert  and  Johnson  [12]  distinguish  product  organization  complexity,  product  function  complexity, 
product  operational  complexity,  development  process  complexity,  and  development  organization 
complexity.  The  paper  describes  a  variety  of  aspects  of  complexity  including  structure,  uncertainty 
(knowledge  in  hand  versus  sufficient  knowledge  for  design,  evaluation  and  operation),  dynamic 
cascading  effects,  and  dynamic  adaptation.  The  goal  of  the  paper  is  to  advance  understanding  of  design 
processes  and  design  outcomes.  No  metrics  are  given.  No  direct  relationship  to  acquisition  risk  is 
presented. 
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